Table of Contents

Keystone - distributed on WAN

Keystone Cluster Setup

WAN Keystone Setup

    # host keystone.ha.infn.it
    keystone.ha.infn.it has address 90.147.102.133
    keystone.ha.infn.it has address 90.147.112.4   
    keystone.ha.infn.it has address 193.206.210.230

Keystone + HAProxy Setup - Installation & Configuration

Open Ports

PD setup & logs

Keystone with SSL support

Taking as a starting point Ref. 6 & 7.

  1. creation of a "fake", auto-signed certificate:
    # /usr/lib/ssl/misc/CA.pl -newca           
    # /usr/lib/ssl/misc/CA.pl -newreq
    # /usr/lib/ssl/misc/CA.pl -sign
    # openssl rsa -in newkey.pem -out newkey_ue.pem 
  2. copy certificates on all keystone nodes:
    # cp cacert.pem /etc/keystone/ssl/certs/cacert.pem
    # cp newcert.pem /etc/keystone/ssl/certs/newcert.pem
    # cp newkey_ue.pem /etc/keystone/ssl/private/newkey.pem 
     
    # chmod 400 /etc/keystone/ssl/private/newkey.pem
    # chown -R keystone:keystone /etc/keystone 
  3. modify keystone.conf to enable ssl:
    # vi /etc/keystone/keystone.conf 
    [...]
    [ssl]
    enable = True
    certfile = /etc/keystone/ssl/certs/newcert.pem
    keyfile = /etc/keystone/ssl/private/newkey.pem
    ca_certs = /etc/keystone/ssl/certs/cacert.pem
    cert_required = False 
    [...]
  4. restart keystone:
    # service keystone restart 
  5. on haproxy nodes, change /etc/haproxy/haproxy.conf:
    <     mode http
    <     option httplog
    ---
    > #    mode http
    >     mode tcp
    > #    option httplog
     
    30c31
    <     option httpchk
    ---
    > #    option httpchk
     
    40c41
    <     option httpchk
    ---
    > #    option httpchk
     
    48c49
    <     stats enable
    ---
    > #    stats enable 
  6. restart haproxy:
    # haproxy -f /etc/haproxy/haproxy.cf
  7. some tests:
    # strace -e connect keystone --os_auth_url https://keystone.ha.infn.it:5000/v2.0 --os_username nagios --os_tenant_name demo --os_password XXXXXXXXX tenant-list
    --- SIGCHLD (Child exited) @ 0 (0) ---
    --- SIGCHLD (Child exited) @ 0 (0) ---
    connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
    connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
    connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("193.206.210.147")}, 16) = 0
    connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("193.206.210.147")}, 16) = 0
    connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("193.206.210.147")}, 16) = 0
    connect(3, {sa_family=AF_INET, sin_port=htons(5000), sin_addr=inet_addr("90.147.112.4")}, 16) = 0
    connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
    connect(3, {sa_family=AF_INET, sin_port=htons(5000), sin_addr=inet_addr("193.206.210.230")}, 16) = 0
    connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
    connect(3, {sa_family=AF_INET, sin_port=htons(5000), sin_addr=inet_addr("90.147.102.133")}, 16) = 0
    connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("193.206.210.147")}, 16) = 0
    connect(4, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("193.206.210.230")}, 16) = 0
    connect(4, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
    connect(4, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("90.147.102.133")}, 16) = 0
    connect(4, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
    connect(4, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("90.147.112.4")}, 16) = 0
    connect(3, {sa_family=AF_INET, sin_port=htons(5000), sin_addr=inet_addr("193.206.210.230")}, 16) = 0
    No handlers could be found for logger "keystoneclient.v2_0.client"
    +----------------------------------+-------------+---------+
    |                id                |     name    | enabled |
    +----------------------------------+-------------+---------+
    | 0239ea1049444a87987ce4275e0aac8f | testreplica | True    |
    | ae8ccba0393b4038b50d590a90df94a8 | demo        | True    |
    +----------------------------------+-------------+---------+
     
    # keystone --endpoint https://keystone.ha.infn.it:35357/v2.0 --token XXXXXXXXX tenant-list
    +----------------------------------+-------------+---------+
    |                id                |     name    | enabled |
    +----------------------------------+-------------+---------+
    | 0239ea1049444a87987ce4275e0aac8f | testreplica | True    |
    | 3d8c9bb15e084bbaa49d4e8af0162a06 | pdtest      | True    |
    | a025708e981e40389a0d2e76e4709ebb | service     | True    |
    | ae8ccba0393b4038b50d590a90df94a8 | demo        | True    |
    +----------------------------------+-------------+---------+
    # keystone --endpoint https://keystone.ha.infn.it:35357/v2.0 --token XXXXXXXX user-list
    +----------------------------------+---------+-------------------------------+--------+
    |                id                | enabled |             email             |  name  |
    +----------------------------------+---------+-------------------------------+--------+
    | 924293d12b824286a4aef61c909ccc11 | True    | stefano.stalio@lngs.infn.it   | nagios |
    | a92f4071351d4d8b8db84b3834445d0d | True    | cristina.aiftimiei@pd.infn.it | caifti |
    | c32d5b9e3b274734972134152fa829c4 | True    | stefano.stalio@lngs.infn.it   | stalio |
    | d7d369c91269494b804192248dab6632 | True    | marica.antonacci@ba.infn.it   | marica |
    +----------------------------------+---------+-------------------------------+--------+

Percona/MySQL with SSL support

Following Ref. 8 we executed:

  1. copy on each keystone node the "fake" cert.pem & key.pem provided by StefanoS:
    # mkdir -p /var/lib/mysql/certs
    # mv *.pem /var/lib/mysql/certs/
    # chown mysql:mysql /var/lib/mysql/certs/*
    # chmod 644 /var/lib/mysql/certs/cert.pem
    # chmod 400 /var/lib/mysql/certs/key.pem
  2. modify my.conf file, to add new param wsrep_provider_options:
    # grep wsrep_provider_options /etc/mysql/my.cnf 
    wsrep_provider_options="socket.ssl_cert=/var/lib//mysql/certs/cert.pem; socket.ssl_key=/var/lib/mysql/certs/key.pem"
  3. stop mysql on all nodes:
    <ba>   # service mysql stop
    <lngs> # service mysql stop
    <pd>   # service mysql stop
  4. on <pd> node change wsrep_cluster_address in my.cnf:
    # sed -i \
    -e "s/#wsrep_cluster_address=gcomm:\/\/$/wsrep_cluster_address=gcomm:\/\//" \ 
    -e "s/wsrep_cluster_address=gcomm:\/\/193.206.210.164,90.147.112.3,90.147.102.131/#wsrep_cluster_address=gcomm:\/\/193.206.210.164,90.147.112.3,90.147.102.131/" /etc/mysql/my.cnf
  5. start mysql on all nodes:
    <pd>   # service mysql start
    <lngs> # service mysql start
    <ba>   # service mysql start
  6. on <pd> node change back wsrep_cluster_address in my.cnf:
    # sed -i \
    -e "s/wsrep_cluster_address=gcomm:\/\/$/#wsrep_cluster_address=gcomm:\/\//" \ 
    -e "s/#wsrep_cluster_address=gcomm:\/\/193.206.210.164,90.147.112.3,90.147.102.131/wsrep_cluster_address=gcomm:\/\/193.206.210.164,90.147.112.3,90.147.102.131/" /etc/mysql/my.cnf
  7. after few secs on the log everything is ok:
    # tail -f /var/lib/mysql/cream-11.err
    140116 17:35:07 [Note] WSREP: SSL handshake successful, remote endpoint ssl://90.147.102.131:4567 local endpoint ssl://193.206.210.164:37840 cipher: AES128-SHA compression: 
    140116 17:35:08 [Note] WSREP: SSL handshake successful, remote endpoint ssl://90.147.112.3:56527 local endpoint ssl://193.206.210.164:4567 cipher: AES128-SHA compression:
    [...]
    140116 17:35:08 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 29842596-7ecc-11e3-9622-4f94cb3a9ab0
    140116 17:35:08 [Note] WSREP: STATE EXCHANGE: sent state msg: 29842596-7ecc-11e3-9622-4f94cb3a9ab0
    140116 17:35:08 [Note] WSREP: STATE EXCHANGE: got state msg: 29842596-7ecc-11e3-9622-4f94cb3a9ab0 from 0 (cream-11)
    140116 17:35:08 [Note] WSREP: STATE EXCHANGE: got state msg: 29842596-7ecc-11e3-9622-4f94cb3a9ab0 from 1 (keystone-infn)
    140116 17:35:08 [Note] WSREP: STATE EXCHANGE: got state msg: 29842596-7ecc-11e3-9622-4f94cb3a9ab0 from 2 (keystone2)
    140116 17:35:08 [Note] WSREP: Quorum results:
            version    = 2,
            component  = PRIMARY,
            conf_id    = 28,
            members    = 3/3 (joined/total),
            act_id     = 1857,
            last_appl. = 0,
            protocols  = 0/4/2 (gcs/repl/appl),
            group UUID = 6e12b786-6316-11e3-be17-57dea5f4d77d
    [...]
  8. some tests
    • status before tests:
      mysql(PD)> use percona;
      [...]
      mysql(PD)> show tables;
      +-------------------+
      | Tables_in_percona |
      +-------------------+
      | example           |
      +-------------------+
      1 row in set (0.00 sec)
    • new table is created at <ba>, query DB at <pd>:
      mysql(PD)> show tables;
      +-------------------+
      | Tables_in_percona |
      +-------------------+
      | example           |
      | test              |
      +-------------------+
      2 rows in set (0.00 sec)
    • new values are inserted at <lngs>, query DB at <pd>:
      mysql(LNGS)>use percona;
      [...]
      mysql(LNGS)>insert into example (node_id, node_name) values (2, 'ssl');
      [...]
       
      mysql(PD)> select * from example;
      +---------+-----------+
      | node_id | node_name |
      +---------+-----------+
      |       1 | percona1  |
      |       2 | ssl       |
      +---------+-----------+
      2 rows in set (0.00 sec)