====== Keystone - distributed on WAN ====== ===== Reference Links ===== - {{:progetti:cloud-areapd:keystone-setup-info.pdf|Keystone - setup info}} - [[http://docs.openstack.org/grizzly/openstack-compute/install/apt/content/install-keystone.html|Grizzly - Install Keystone]] - [[http://www.percona.com/software/percona-xtradb-cluster|Percona XtraDB Cluster]] - [[http://haproxy.1wt.eu/|HAProxy]] - [[http://indico.cern.ch/getFile.py/access?contribId=44&sessionId=2&resId=1&materialId=slides&confId=220443|DNA HA - A multimaster DNS configuration for Disaster Recovery and Business Continuity environment]] - [[http://docs.openstack.org/developer/keystone/configuration.html#ssl|Keystone SSL Configuration (1)]] - [[http://docs.openstack.org/grizzly/openstack-compute/admin/content//keystone-ssl.html|Keystone SSL Configuration (2)]] - [[http://sql.dzone.com/articles/percona-xtradb-cluster-mysql|Percona XtraDB Cluster for MySQL and encrypted Galera replication]] ===== Keystone Cluster Setup ===== {{ :cn:ccr:cloud:autenticazione_openstack:indios-fig3.png?400 |WAN Keystone Setup}} * Keystone servers: * keystone 1: keystone.pd.infn.it - 193.206.210.164 * keystone 2: keystone-infn.lngs.infn.it - 90.147.112.3 * keystone 3: keystone2.cloud.ba.infn.it - 90.147.102.131 * HAproxyes * ha-proxy 1: haproxypd.pd.infn.it - 193.206.210.230 * ha-proxy 2: keystone-infn-haproxy.lngs.infn.it - 90.147.112.4 * ha-proxy 3: haproxy.cloud.ba.infn.it - 90.147.102.133 * Alias for the proxy-nodes - **keystone.ha.infn.it**: # host keystone.ha.infn.it keystone.ha.infn.it has address 90.147.102.133 keystone.ha.infn.it has address 90.147.112.4 keystone.ha.infn.it has address 193.206.210.230 ===== Keystone + HAProxy Setup - Installation & Configuration ===== ==== Open Ports ==== * on each keystone node: * 3306, 4444, 4567, 4568 - open to the other 2 keystone-nodes * 5000, 35357 - open to haproxy-nodes * on each haproxy-node: * 5000, 35357 - open for "outside" ==== PD setup & logs ==== * cream-mstr-017 / 193.206.210.47 - KVM master * CPU - (8) Intel Xeon E5420 2500 * RAM - 16GB * Disk - 150GB SAS * cream-11 / 193.206.210.164 - keystone (primary server) * Ubuntu 12.04 * CPU - 2 * RAM - 2GB * Disk: vda1 - 20G, vdb1 - 40G * cert-15 / 193.206.210.164 - haproxy * Ubuntu 12.04 * CPU - 2 * RAM - 2GB * Disk: vda1 - 20G, vdb1 - 40G * Configuration log for keystone-server (cream-11) & for haproxy (cert-15): {{:cn:ccr:cloud:autenticazione_openstack:keystone_geografico_pd.txt|}} * Summary of some tests: {{:cn:ccr:cloud:autenticazione_openstack:keystone_distribuito_tests.txt|}} ===== Keystone with SSL support ===== Taking as a starting point [[cn:ccr:cloud:autenticazione_openstack:keystone_wan#reference_links|Ref. 6 & 7]]. - creation of a "fake", auto-signed certificate: # /usr/lib/ssl/misc/CA.pl -newca # /usr/lib/ssl/misc/CA.pl -newreq # /usr/lib/ssl/misc/CA.pl -sign # openssl rsa -in newkey.pem -out newkey_ue.pem - copy certificates on all keystone nodes: # cp cacert.pem /etc/keystone/ssl/certs/cacert.pem # cp newcert.pem /etc/keystone/ssl/certs/newcert.pem # cp newkey_ue.pem /etc/keystone/ssl/private/newkey.pem # chmod 400 /etc/keystone/ssl/private/newkey.pem # chown -R keystone:keystone /etc/keystone - modify //keystone.conf// to enable ssl: # vi /etc/keystone/keystone.conf [...] [ssl] enable = True certfile = /etc/keystone/ssl/certs/newcert.pem keyfile = /etc/keystone/ssl/private/newkey.pem ca_certs = /etc/keystone/ssl/certs/cacert.pem cert_required = False [...] - restart keystone: # service keystone restart - on haproxy nodes, change ///etc/haproxy/haproxy.conf//: < mode http < option httplog --- > # mode http > mode tcp > # option httplog 30c31 < option httpchk --- > # option httpchk 40c41 < option httpchk --- > # option httpchk 48c49 < stats enable --- > # stats enable - restart haproxy: # haproxy -f /etc/haproxy/haproxy.cf - some tests: # strace -e connect keystone --os_auth_url https://keystone.ha.infn.it:5000/v2.0 --os_username nagios --os_tenant_name demo --os_password XXXXXXXXX tenant-list --- SIGCHLD (Child exited) @ 0 (0) --- --- SIGCHLD (Child exited) @ 0 (0) --- connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("193.206.210.147")}, 16) = 0 connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("193.206.210.147")}, 16) = 0 connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("193.206.210.147")}, 16) = 0 connect(3, {sa_family=AF_INET, sin_port=htons(5000), sin_addr=inet_addr("90.147.112.4")}, 16) = 0 connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0 connect(3, {sa_family=AF_INET, sin_port=htons(5000), sin_addr=inet_addr("193.206.210.230")}, 16) = 0 connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0 connect(3, {sa_family=AF_INET, sin_port=htons(5000), sin_addr=inet_addr("90.147.102.133")}, 16) = 0 connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("193.206.210.147")}, 16) = 0 connect(4, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("193.206.210.230")}, 16) = 0 connect(4, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0 connect(4, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("90.147.102.133")}, 16) = 0 connect(4, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0 connect(4, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("90.147.112.4")}, 16) = 0 connect(3, {sa_family=AF_INET, sin_port=htons(5000), sin_addr=inet_addr("193.206.210.230")}, 16) = 0 No handlers could be found for logger "keystoneclient.v2_0.client" +----------------------------------+-------------+---------+ | id | name | enabled | +----------------------------------+-------------+---------+ | 0239ea1049444a87987ce4275e0aac8f | testreplica | True | | ae8ccba0393b4038b50d590a90df94a8 | demo | True | +----------------------------------+-------------+---------+ # keystone --endpoint https://keystone.ha.infn.it:35357/v2.0 --token XXXXXXXXX tenant-list +----------------------------------+-------------+---------+ | id | name | enabled | +----------------------------------+-------------+---------+ | 0239ea1049444a87987ce4275e0aac8f | testreplica | True | | 3d8c9bb15e084bbaa49d4e8af0162a06 | pdtest | True | | a025708e981e40389a0d2e76e4709ebb | service | True | | ae8ccba0393b4038b50d590a90df94a8 | demo | True | +----------------------------------+-------------+---------+ # keystone --endpoint https://keystone.ha.infn.it:35357/v2.0 --token XXXXXXXX user-list +----------------------------------+---------+-------------------------------+--------+ | id | enabled | email | name | +----------------------------------+---------+-------------------------------+--------+ | 924293d12b824286a4aef61c909ccc11 | True | stefano.stalio@lngs.infn.it | nagios | | a92f4071351d4d8b8db84b3834445d0d | True | cristina.aiftimiei@pd.infn.it | caifti | | c32d5b9e3b274734972134152fa829c4 | True | stefano.stalio@lngs.infn.it | stalio | | d7d369c91269494b804192248dab6632 | True | marica.antonacci@ba.infn.it | marica | +----------------------------------+---------+-------------------------------+--------+ ===== Percona/MySQL with SSL support ===== Following [[http://sql.dzone.com/articles/percona-xtradb-cluster-mysql|Ref. 8]] we executed: - copy on each keystone node the "fake" cert.pem & key.pem provided by StefanoS: # mkdir -p /var/lib/mysql/certs # mv *.pem /var/lib/mysql/certs/ # chown mysql:mysql /var/lib/mysql/certs/* # chmod 644 /var/lib/mysql/certs/cert.pem # chmod 400 /var/lib/mysql/certs/key.pem - modify my.conf file, to add new param //wsrep_provider_options//: # grep wsrep_provider_options /etc/mysql/my.cnf wsrep_provider_options="socket.ssl_cert=/var/lib//mysql/certs/cert.pem; socket.ssl_key=/var/lib/mysql/certs/key.pem" - stop mysql on all nodes: # service mysql stop # service mysql stop # service mysql stop - on node change //wsrep_cluster_address// in my.cnf: # sed -i \ -e "s/#wsrep_cluster_address=gcomm:\/\/$/wsrep_cluster_address=gcomm:\/\//" \ -e "s/wsrep_cluster_address=gcomm:\/\/193.206.210.164,90.147.112.3,90.147.102.131/#wsrep_cluster_address=gcomm:\/\/193.206.210.164,90.147.112.3,90.147.102.131/" /etc/mysql/my.cnf - start mysql on all nodes: # service mysql start # service mysql start # service mysql start - on node change **back** //wsrep_cluster_address// in my.cnf: # sed -i \ -e "s/wsrep_cluster_address=gcomm:\/\/$/#wsrep_cluster_address=gcomm:\/\//" \ -e "s/#wsrep_cluster_address=gcomm:\/\/193.206.210.164,90.147.112.3,90.147.102.131/wsrep_cluster_address=gcomm:\/\/193.206.210.164,90.147.112.3,90.147.102.131/" /etc/mysql/my.cnf - after few secs on the log everything is ok: # tail -f /var/lib/mysql/cream-11.err 140116 17:35:07 [Note] WSREP: SSL handshake successful, remote endpoint ssl://90.147.102.131:4567 local endpoint ssl://193.206.210.164:37840 cipher: AES128-SHA compression: 140116 17:35:08 [Note] WSREP: SSL handshake successful, remote endpoint ssl://90.147.112.3:56527 local endpoint ssl://193.206.210.164:4567 cipher: AES128-SHA compression: [...] 140116 17:35:08 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 29842596-7ecc-11e3-9622-4f94cb3a9ab0 140116 17:35:08 [Note] WSREP: STATE EXCHANGE: sent state msg: 29842596-7ecc-11e3-9622-4f94cb3a9ab0 140116 17:35:08 [Note] WSREP: STATE EXCHANGE: got state msg: 29842596-7ecc-11e3-9622-4f94cb3a9ab0 from 0 (cream-11) 140116 17:35:08 [Note] WSREP: STATE EXCHANGE: got state msg: 29842596-7ecc-11e3-9622-4f94cb3a9ab0 from 1 (keystone-infn) 140116 17:35:08 [Note] WSREP: STATE EXCHANGE: got state msg: 29842596-7ecc-11e3-9622-4f94cb3a9ab0 from 2 (keystone2) 140116 17:35:08 [Note] WSREP: Quorum results: version = 2, component = PRIMARY, conf_id = 28, members = 3/3 (joined/total), act_id = 1857, last_appl. = 0, protocols = 0/4/2 (gcs/repl/appl), group UUID = 6e12b786-6316-11e3-be17-57dea5f4d77d [...] - some tests * status before tests: mysql(PD)> use percona; [...] mysql(PD)> show tables; +-------------------+ | Tables_in_percona | +-------------------+ | example | +-------------------+ 1 row in set (0.00 sec) * new table is created at , query DB at : mysql(PD)> show tables; +-------------------+ | Tables_in_percona | +-------------------+ | example | | test | +-------------------+ 2 rows in set (0.00 sec) * new values are inserted at , query DB at : mysql(LNGS)>use percona; [...] mysql(LNGS)>insert into example (node_id, node_name) values (2, 'ssl'); [...] mysql(PD)> select * from example; +---------+-----------+ | node_id | node_name | +---------+-----------+ | 1 | percona1 | | 2 | ssl | +---------+-----------+ 2 rows in set (0.00 sec)