INFN wiki

User Tools

Site Tools

Trace: • Keystone - distributed on WAN
cn:ccr:cloud:autenticazione_openstack:keystone_wan

Table of Contents

Keystone - distributed on WAN

  1. Keystone - setup info
  2. Grizzly - Install Keystone
  3. Percona XtraDB Cluster
  4. HAProxy
  5. DNA HA - A multimaster DNS configuration for Disaster Recovery and Business Continuity environment
  6. Keystone SSL Configuration (1)
  7. Keystone SSL Configuration (2)
  8. Percona XtraDB Cluster for MySQL and encrypted Galera replication

Keystone Cluster Setup

WAN Keystone Setup

  • Keystone servers:
    • keystone 1: keystone.pd.infn.it - 193.206.210.164
    • keystone 2: keystone-infn.lngs.infn.it - 90.147.112.3
    • keystone 3: keystone2.cloud.ba.infn.it - 90.147.102.131
  • HAproxyes
    • ha-proxy 1: haproxypd.pd.infn.it - 193.206.210.230
    • ha-proxy 2: keystone-infn-haproxy.lngs.infn.it - 90.147.112.4
    • ha-proxy 3: haproxy.cloud.ba.infn.it - 90.147.102.133
  • Alias for the proxy-nodes - keystone.ha.infn.it:
    # host keystone.ha.infn.it
    keystone.ha.infn.it has address 90.147.102.133
    keystone.ha.infn.it has address 90.147.112.4   
    keystone.ha.infn.it has address 193.206.210.230

Keystone + HAProxy Setup - Installation & Configuration

Open Ports

  • on each keystone node:
    • 3306, 4444, 4567, 4568 - open to the other 2 keystone-nodes
    • 5000, 35357 - open to haproxy-nodes
  • on each haproxy-node:
    • 5000, 35357 - open for "outside"

PD setup & logs

  • cream-mstr-017 / 193.206.210.47 - KVM master
    • CPU - (8) Intel Xeon E5420 2500
    • RAM - 16GB
    • Disk - 150GB SAS
  • cream-11 / 193.206.210.164 - keystone (primary server)
    • Ubuntu 12.04
    • CPU - 2
    • RAM - 2GB
    • Disk: vda1 - 20G, vdb1 - 40G
  • cert-15 / 193.206.210.164 - haproxy
    • Ubuntu 12.04
    • CPU - 2
    • RAM - 2GB
    • Disk: vda1 - 20G, vdb1 - 40G

Keystone with SSL support

Taking as a starting point Ref. 6 & 7.

  1. creation of a "fake", auto-signed certificate:
    # /usr/lib/ssl/misc/CA.pl -newca           
# /usr/lib/ssl/misc/CA.pl -newreq
# /usr/lib/ssl/misc/CA.pl -sign
# openssl rsa -in newkey.pem -out newkey_ue.pem
  2. copy certificates on all keystone nodes: 
    # cp cacert.pem /etc/keystone/ssl/certs/cacert.pem
# cp newcert.pem /etc/keystone/ssl/certs/newcert.pem
# cp newkey_ue.pem /etc/keystone/ssl/private/newkey.pem 
 
# chmod 400 /etc/keystone/ssl/private/newkey.pem
# chown -R keystone:keystone /etc/keystone
  3. modify keystone.conf to enable ssl: 
    # vi /etc/keystone/keystone.conf 
[...]
[ssl]
enable = True
certfile = /etc/keystone/ssl/certs/newcert.pem
keyfile = /etc/keystone/ssl/private/newkey.pem
ca_certs = /etc/keystone/ssl/certs/cacert.pem
cert_required = False 
[...]
  4. restart keystone: 
    # service keystone restart
  5. on haproxy nodes, change /etc/haproxy/haproxy.conf: 
    <     mode http
<     option httplog
---
> #    mode http
>     mode tcp
> #    option httplog
 
30c31
<     option httpchk
---
> #    option httpchk
 
40c41
<     option httpchk
---
> #    option httpchk
 
48c49
<     stats enable
---
> #    stats enable
  6. restart haproxy:
    # haproxy -f /etc/haproxy/haproxy.cf
  7. some tests: 
    # strace -e connect keystone --os_auth_url https://keystone.ha.infn.it:5000/v2.0 --os_username nagios --os_tenant_name demo --os_password XXXXXXXXX tenant-list
--- SIGCHLD (Child exited) @ 0 (0) ---
--- SIGCHLD (Child exited) @ 0 (0) ---
connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("193.206.210.147")}, 16) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("193.206.210.147")}, 16) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("193.206.210.147")}, 16) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(5000), sin_addr=inet_addr("90.147.112.4")}, 16) = 0
connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(5000), sin_addr=inet_addr("193.206.210.230")}, 16) = 0
connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(5000), sin_addr=inet_addr("90.147.102.133")}, 16) = 0
connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("193.206.210.147")}, 16) = 0
connect(4, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("193.206.210.230")}, 16) = 0
connect(4, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
connect(4, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("90.147.102.133")}, 16) = 0
connect(4, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
connect(4, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("90.147.112.4")}, 16) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(5000), sin_addr=inet_addr("193.206.210.230")}, 16) = 0
No handlers could be found for logger "keystoneclient.v2_0.client"
+----------------------------------+-------------+---------+
|                id                |     name    | enabled |
+----------------------------------+-------------+---------+
| 0239ea1049444a87987ce4275e0aac8f | testreplica | True    |
| ae8ccba0393b4038b50d590a90df94a8 | demo        | True    |
+----------------------------------+-------------+---------+
 
# keystone --endpoint https://keystone.ha.infn.it:35357/v2.0 --token XXXXXXXXX tenant-list
+----------------------------------+-------------+---------+
|                id                |     name    | enabled |
+----------------------------------+-------------+---------+
| 0239ea1049444a87987ce4275e0aac8f | testreplica | True    |
| 3d8c9bb15e084bbaa49d4e8af0162a06 | pdtest      | True    |
| a025708e981e40389a0d2e76e4709ebb | service     | True    |
| ae8ccba0393b4038b50d590a90df94a8 | demo        | True    |
+----------------------------------+-------------+---------+
# keystone --endpoint https://keystone.ha.infn.it:35357/v2.0 --token XXXXXXXX user-list
+----------------------------------+---------+-------------------------------+--------+
|                id                | enabled |             email             |  name  |
+----------------------------------+---------+-------------------------------+--------+
| 924293d12b824286a4aef61c909ccc11 | True    | stefano.stalio@lngs.infn.it   | nagios |
| a92f4071351d4d8b8db84b3834445d0d | True    | cristina.aiftimiei@pd.infn.it | caifti |
| c32d5b9e3b274734972134152fa829c4 | True    | stefano.stalio@lngs.infn.it   | stalio |
| d7d369c91269494b804192248dab6632 | True    | marica.antonacci@ba.infn.it   | marica |
+----------------------------------+---------+-------------------------------+--------+

Percona/MySQL with SSL support

Following Ref. 8 we executed:

  1. copy on each keystone node the "fake" cert.pem & key.pem provided by StefanoS:
    # mkdir -p /var/lib/mysql/certs
# mv *.pem /var/lib/mysql/certs/
# chown mysql:mysql /var/lib/mysql/certs/*
# chmod 644 /var/lib/mysql/certs/cert.pem
# chmod 400 /var/lib/mysql/certs/key.pem
  2. modify my.conf file, to add new param wsrep_provider_options:
    # grep wsrep_provider_options /etc/mysql/my.cnf 
wsrep_provider_options="socket.ssl_cert=/var/lib//mysql/certs/cert.pem; socket.ssl_key=/var/lib/mysql/certs/key.pem"
  3. stop mysql on all nodes:
    <ba>   # service mysql stop
<lngs> # service mysql stop
<pd>   # service mysql stop
  4. on <pd> node change wsrep_cluster_address in my.cnf:
    # sed -i \
-e "s/#wsrep_cluster_address=gcomm:\/\/$/wsrep_cluster_address=gcomm:\/\//" \ 
-e "s/wsrep_cluster_address=gcomm:\/\/193.206.210.164,90.147.112.3,90.147.102.131/#wsrep_cluster_address=gcomm:\/\/193.206.210.164,90.147.112.3,90.147.102.131/" /etc/mysql/my.cnf
  5. start mysql on all nodes:
    <pd>   # service mysql start
<lngs> # service mysql start
<ba>   # service mysql start
  6. on <pd> node change back wsrep_cluster_address in my.cnf:
    # sed -i \
-e "s/wsrep_cluster_address=gcomm:\/\/$/#wsrep_cluster_address=gcomm:\/\//" \ 
-e "s/#wsrep_cluster_address=gcomm:\/\/193.206.210.164,90.147.112.3,90.147.102.131/wsrep_cluster_address=gcomm:\/\/193.206.210.164,90.147.112.3,90.147.102.131/" /etc/mysql/my.cnf
  7. after few secs on the log everything is ok:
    # tail -f /var/lib/mysql/cream-11.err
140116 17:35:07 [Note] WSREP: SSL handshake successful, remote endpoint ssl://90.147.102.131:4567 local endpoint ssl://193.206.210.164:37840 cipher: AES128-SHA compression: 
140116 17:35:08 [Note] WSREP: SSL handshake successful, remote endpoint ssl://90.147.112.3:56527 local endpoint ssl://193.206.210.164:4567 cipher: AES128-SHA compression:
[...]
140116 17:35:08 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 29842596-7ecc-11e3-9622-4f94cb3a9ab0
140116 17:35:08 [Note] WSREP: STATE EXCHANGE: sent state msg: 29842596-7ecc-11e3-9622-4f94cb3a9ab0
140116 17:35:08 [Note] WSREP: STATE EXCHANGE: got state msg: 29842596-7ecc-11e3-9622-4f94cb3a9ab0 from 0 (cream-11)
140116 17:35:08 [Note] WSREP: STATE EXCHANGE: got state msg: 29842596-7ecc-11e3-9622-4f94cb3a9ab0 from 1 (keystone-infn)
140116 17:35:08 [Note] WSREP: STATE EXCHANGE: got state msg: 29842596-7ecc-11e3-9622-4f94cb3a9ab0 from 2 (keystone2)
140116 17:35:08 [Note] WSREP: Quorum results:
        version    = 2,
        component  = PRIMARY,
        conf_id    = 28,
        members    = 3/3 (joined/total),
        act_id     = 1857,
        last_appl. = 0,
        protocols  = 0/4/2 (gcs/repl/appl),
        group UUID = 6e12b786-6316-11e3-be17-57dea5f4d77d
[...]
  8. some tests
    • status before tests:
      mysql(PD)> use percona;
[...]
mysql(PD)> show tables;
+-------------------+
| Tables_in_percona |
+-------------------+
| example           |
+-------------------+
1 row in set (0.00 sec)
    • new table is created at <ba>, query DB at <pd>:
      mysql(PD)> show tables;
+-------------------+
| Tables_in_percona |
+-------------------+
| example           |
| test              |
+-------------------+
2 rows in set (0.00 sec)
    • new values are inserted at <lngs>, query DB at <pd>:
      mysql(LNGS)>use percona;
[...]
mysql(LNGS)>insert into example (node_id, node_name) values (2, 'ssl');
[...]
 
mysql(PD)> select * from example;
+---------+-----------+
| node_id | node_name |
+---------+-----------+
|       1 | percona1  |
|       2 | ssl       |
+---------+-----------+
2 rows in set (0.00 sec)
cn/ccr/cloud/autenticazione_openstack/keystone_wan.txt · Last modified: 2014/05/14 19:40 by aiftim@infn.it
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki