INFN wiki

User Tools

Site Tools

Trace: • FARM DI ANALISI INTERATTIVA(FAI) • Gruppo di Attività di INFN Bo su (Deep) Machine Learning per la microelettronica • ICARUS at CNAF • Istruzioni operative • pcftkvmeX Fresh OS Install - notes
cn:ccr:cloud:autenticazione_openstack:keystone_wan

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
cn:ccr:cloud:autenticazione_openstack:keystone_wan [2014/01/20 15:42] – [PD Keystone + HAProxy Setup] aiftim@infn.itcn:ccr:cloud:autenticazione_openstack:keystone_wan [2014/05/14 19:40] (current) – [Keystone Cluster Setup] aiftim@infn.it
Line 1: Line 1:
 +====== Keystone - distributed on WAN ======
  
 +===== Reference Links =====
 +  - {{:progetti:cloud-areapd:keystone-setup-info.pdf|Keystone - setup info}}
 +  - [[http://docs.openstack.org/grizzly/openstack-compute/install/apt/content/install-keystone.html|Grizzly - Install Keystone]]
 +  - [[http://www.percona.com/software/percona-xtradb-cluster|Percona XtraDB Cluster]]
 +  - [[http://haproxy.1wt.eu/|HAProxy]]
 +  - [[http://indico.cern.ch/getFile.py/access?contribId=44&sessionId=2&resId=1&materialId=slides&confId=220443|DNA HA - A multimaster DNS configuration for Disaster Recovery and Business Continuity environment]]
 +  - [[http://docs.openstack.org/developer/keystone/configuration.html#ssl|Keystone SSL Configuration (1)]]
 +  - [[http://docs.openstack.org/grizzly/openstack-compute/admin/content//keystone-ssl.html|Keystone SSL Configuration (2)]]
 +  - [[http://sql.dzone.com/articles/percona-xtradb-cluster-mysql|Percona XtraDB Cluster for MySQL and encrypted Galera replication]]
 +
 +===== Keystone Cluster Setup =====
 +
 +
 +{{ :cn:ccr:cloud:autenticazione_openstack:indios-fig3.png?400 |WAN Keystone Setup}}
 +
 +  * Keystone servers:
 +    * keystone 1: keystone.pd.infn.it - 193.206.210.164
 +    * keystone 2: keystone-infn.lngs.infn.it - 90.147.112.3
 +    * keystone 3: keystone2.cloud.ba.infn.it - 90.147.102.131
 +  * HAproxyes
 +    * ha-proxy 1: haproxypd.pd.infn.it - 193.206.210.230
 +    * ha-proxy 2: keystone-infn-haproxy.lngs.infn.it - 90.147.112.4
 +    * ha-proxy 3: haproxy.cloud.ba.infn.it - 90.147.102.133
 +
 +  * Alias for the proxy-nodes - **keystone.ha.infn.it**:
 +<code bash>
 +    # host keystone.ha.infn.it
 +    keystone.ha.infn.it has address 90.147.102.133
 +    keystone.ha.infn.it has address 90.147.112.4   
 +    keystone.ha.infn.it has address 193.206.210.230
 +</code> 
 +
 +===== Keystone + HAProxy Setup - Installation & Configuration =====
 +
 +==== Open Ports ====
 +  * on each keystone node:
 +    * 3306, 4444, 4567, 4568 - open to the other 2 keystone-nodes
 +    * 5000, 35357 - open to haproxy-nodes
 +  * on each haproxy-node:
 +    * 5000, 35357 - open for "outside"
 +
 +
 +
 +
 +==== PD setup & logs ====
 +    * cream-mstr-017 / 193.206.210.47 - KVM master
 +      * CPU - (8) Intel Xeon E5420 2500
 +      * RAM - 16GB
 +      * Disk - 150GB SAS
 +    * cream-11 / 193.206.210.164 - keystone (primary server)
 +      * Ubuntu 12.04
 +      * CPU - 2
 +      * RAM - 2GB
 +      * Disk: vda1 - 20G, vdb1 - 40G
 +    * cert-15 / 193.206.210.164 - haproxy
 +      * Ubuntu 12.04
 +      * CPU - 2
 +      * RAM - 2GB
 +      * Disk: vda1 - 20G, vdb1 - 40G
 +
 +
 +  * Configuration log for keystone-server (cream-11) & for haproxy (cert-15): {{:cn:ccr:cloud:autenticazione_openstack:keystone_geografico_pd.txt|}}
 +  * Summary of some tests: {{:cn:ccr:cloud:autenticazione_openstack:keystone_distribuito_tests.txt|}}
 +
 +
 +===== Keystone with SSL support =====
 +
 +Taking as a starting point [[cn:ccr:cloud:autenticazione_openstack:keystone_wan#reference_links|Ref. 6 & 7]].
 +  - creation of a "fake", auto-signed certificate:<code bash>
 +# /usr/lib/ssl/misc/CA.pl -newca           
 +# /usr/lib/ssl/misc/CA.pl -newreq
 +# /usr/lib/ssl/misc/CA.pl -sign
 +# openssl rsa -in newkey.pem -out newkey_ue.pem 
 +</code>
 +  - copy certificates on all keystone nodes: <code bash>
 +# cp cacert.pem /etc/keystone/ssl/certs/cacert.pem
 +# cp newcert.pem /etc/keystone/ssl/certs/newcert.pem
 +# cp newkey_ue.pem /etc/keystone/ssl/private/newkey.pem 
 +
 +# chmod 400 /etc/keystone/ssl/private/newkey.pem
 +# chown -R keystone:keystone /etc/keystone 
 +</code>
 +  - modify //keystone.conf// to enable ssl: <code bash>
 +# vi /etc/keystone/keystone.conf 
 +[...]
 +[ssl]
 +enable = True
 +certfile = /etc/keystone/ssl/certs/newcert.pem
 +keyfile = /etc/keystone/ssl/private/newkey.pem
 +ca_certs = /etc/keystone/ssl/certs/cacert.pem
 +cert_required = False 
 +[...]
 +</code>
 +  - restart keystone: <code bash>
 +# service keystone restart 
 +</code>
 +  - on haproxy nodes, change ///etc/haproxy/haproxy.conf//: <code bash>
 +<     mode http
 +<     option httplog
 +---
 +> #    mode http
 +>     mode tcp
 +> #    option httplog
 +
 +30c31
 +<     option httpchk
 +---
 +> #    option httpchk
 +
 +40c41
 +<     option httpchk
 +---
 +> #    option httpchk
 +
 +48c49
 +<     stats enable
 +---
 +> #    stats enable 
 +</code>
 +  - restart haproxy:<code bash>
 +# haproxy -f /etc/haproxy/haproxy.cf
 +</code>
 +  - some tests: <code bash>
 +# strace -e connect keystone --os_auth_url https://keystone.ha.infn.it:5000/v2.0 --os_username nagios --os_tenant_name demo --os_password XXXXXXXXX tenant-list
 +--- SIGCHLD (Child exited) @ 0 (0) ---
 +--- SIGCHLD (Child exited) @ 0 (0) ---
 +connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
 +connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
 +connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("193.206.210.147")}, 16) = 0
 +connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("193.206.210.147")}, 16) = 0
 +connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("193.206.210.147")}, 16) = 0
 +connect(3, {sa_family=AF_INET, sin_port=htons(5000), sin_addr=inet_addr("90.147.112.4")}, 16) = 0
 +connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
 +connect(3, {sa_family=AF_INET, sin_port=htons(5000), sin_addr=inet_addr("193.206.210.230")}, 16) = 0
 +connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
 +connect(3, {sa_family=AF_INET, sin_port=htons(5000), sin_addr=inet_addr("90.147.102.133")}, 16) = 0
 +connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("193.206.210.147")}, 16) = 0
 +connect(4, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("193.206.210.230")}, 16) = 0
 +connect(4, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
 +connect(4, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("90.147.102.133")}, 16) = 0
 +connect(4, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
 +connect(4, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("90.147.112.4")}, 16) = 0
 +connect(3, {sa_family=AF_INET, sin_port=htons(5000), sin_addr=inet_addr("193.206.210.230")}, 16) = 0
 +No handlers could be found for logger "keystoneclient.v2_0.client"
 ++----------------------------------+-------------+---------+
 +|                id                |     name    | enabled |
 ++----------------------------------+-------------+---------+
 +| 0239ea1049444a87987ce4275e0aac8f | testreplica | True    |
 +| ae8ccba0393b4038b50d590a90df94a8 | demo        | True    |
 ++----------------------------------+-------------+---------+
 +
 +# keystone --endpoint https://keystone.ha.infn.it:35357/v2.0 --token XXXXXXXXX tenant-list
 ++----------------------------------+-------------+---------+
 +|                id                |     name    | enabled |
 ++----------------------------------+-------------+---------+
 +| 0239ea1049444a87987ce4275e0aac8f | testreplica | True    |
 +| 3d8c9bb15e084bbaa49d4e8af0162a06 | pdtest      | True    |
 +| a025708e981e40389a0d2e76e4709ebb | service     | True    |
 +| ae8ccba0393b4038b50d590a90df94a8 | demo        | True    |
 ++----------------------------------+-------------+---------+
 +# keystone --endpoint https://keystone.ha.infn.it:35357/v2.0 --token XXXXXXXX user-list
 ++----------------------------------+---------+-------------------------------+--------+
 +|                id                | enabled |             email             |  name  |
 ++----------------------------------+---------+-------------------------------+--------+
 +| 924293d12b824286a4aef61c909ccc11 | True    | stefano.stalio@lngs.infn.it   | nagios |
 +| a92f4071351d4d8b8db84b3834445d0d | True    | cristina.aiftimiei@pd.infn.it | caifti |
 +| c32d5b9e3b274734972134152fa829c4 | True    | stefano.stalio@lngs.infn.it   | stalio |
 +| d7d369c91269494b804192248dab6632 | True    | marica.antonacci@ba.infn.it   | marica |
 ++----------------------------------+---------+-------------------------------+--------+
 +
 +</code>
 +===== Percona/MySQL with SSL support =====
 +
 +Following [[http://sql.dzone.com/articles/percona-xtradb-cluster-mysql|Ref. 8]] we executed:
 +
 +   - copy on each keystone node the "fake" cert.pem & key.pem provided by StefanoS:<code bash>
 +# mkdir -p /var/lib/mysql/certs
 +# mv *.pem /var/lib/mysql/certs/
 +# chown mysql:mysql /var/lib/mysql/certs/*
 +# chmod 644 /var/lib/mysql/certs/cert.pem
 +# chmod 400 /var/lib/mysql/certs/key.pem
 +</code>
 +  - modify my.conf file, to add new param //wsrep_provider_options//:<code bash>
 +# grep wsrep_provider_options /etc/mysql/my.cnf 
 +wsrep_provider_options="socket.ssl_cert=/var/lib//mysql/certs/cert.pem; socket.ssl_key=/var/lib/mysql/certs/key.pem"
 +</code>
 +  - stop mysql on all nodes:<code bash>
 +<ba>   # service mysql stop
 +<lngs> # service mysql stop
 +<pd>   # service mysql stop
 +</code>
 +  - on <pd> node change //wsrep_cluster_address// in my.cnf:<code bash>
 +# sed -i \
 +-e "s/#wsrep_cluster_address=gcomm:\/\/$/wsrep_cluster_address=gcomm:\/\//" \ 
 +-e "s/wsrep_cluster_address=gcomm:\/\/193.206.210.164,90.147.112.3,90.147.102.131/#wsrep_cluster_address=gcomm:\/\/193.206.210.164,90.147.112.3,90.147.102.131/" /etc/mysql/my.cnf
 +</code>
 +  - start mysql on all nodes:<code bash>
 +<pd>   # service mysql start
 +<lngs> # service mysql start
 +<ba>   # service mysql start
 +</code>
 +  - on <pd> node change **back** //wsrep_cluster_address// in my.cnf:<code bash>
 +# sed -i \
 +-e "s/wsrep_cluster_address=gcomm:\/\/$/#wsrep_cluster_address=gcomm:\/\//" \ 
 +-e "s/#wsrep_cluster_address=gcomm:\/\/193.206.210.164,90.147.112.3,90.147.102.131/wsrep_cluster_address=gcomm:\/\/193.206.210.164,90.147.112.3,90.147.102.131/" /etc/mysql/my.cnf
 +</code>
 +  - after few secs on the log everything is ok:<code bash>
 +# tail -f /var/lib/mysql/cream-11.err
 +140116 17:35:07 [Note] WSREP: SSL handshake successful, remote endpoint ssl://90.147.102.131:4567 local endpoint ssl://193.206.210.164:37840 cipher: AES128-SHA compression: 
 +140116 17:35:08 [Note] WSREP: SSL handshake successful, remote endpoint ssl://90.147.112.3:56527 local endpoint ssl://193.206.210.164:4567 cipher: AES128-SHA compression:
 +[...]
 +140116 17:35:08 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 29842596-7ecc-11e3-9622-4f94cb3a9ab0
 +140116 17:35:08 [Note] WSREP: STATE EXCHANGE: sent state msg: 29842596-7ecc-11e3-9622-4f94cb3a9ab0
 +140116 17:35:08 [Note] WSREP: STATE EXCHANGE: got state msg: 29842596-7ecc-11e3-9622-4f94cb3a9ab0 from 0 (cream-11)
 +140116 17:35:08 [Note] WSREP: STATE EXCHANGE: got state msg: 29842596-7ecc-11e3-9622-4f94cb3a9ab0 from 1 (keystone-infn)
 +140116 17:35:08 [Note] WSREP: STATE EXCHANGE: got state msg: 29842596-7ecc-11e3-9622-4f94cb3a9ab0 from 2 (keystone2)
 +140116 17:35:08 [Note] WSREP: Quorum results:
 +        version    = 2,
 +        component  = PRIMARY,
 +        conf_id    = 28,
 +        members    = 3/3 (joined/total),
 +        act_id     = 1857,
 +        last_appl. = 0,
 +        protocols  = 0/4/2 (gcs/repl/appl),
 +        group UUID = 6e12b786-6316-11e3-be17-57dea5f4d77d
 +[...]
 +</code>
 +  - some tests
 +    * status before tests:<code bash>
 +mysql(PD)> use percona;
 +[...]
 +mysql(PD)> show tables;
 ++-------------------+
 +| Tables_in_percona |
 ++-------------------+
 +| example           |
 ++-------------------+
 +1 row in set (0.00 sec)
 +</code>
 +    * new table is created at <ba>, query DB at <pd>:<code bash>
 +mysql(PD)> show tables;
 ++-------------------+
 +| Tables_in_percona |
 ++-------------------+
 +| example           |
 +| test              |
 ++-------------------+
 +2 rows in set (0.00 sec)
 +</code>
 +    * new values are inserted at <lngs>, query DB at <pd>:<code bash>
 +mysql(LNGS)>use percona;
 +[...]
 +mysql(LNGS)>insert into example (node_id, node_name) values (2, 'ssl');
 +[...]
 +
 +mysql(PD)> select * from example;
 ++---------+-----------+
 +| node_id | node_name |
 ++---------+-----------+
 +|       1 | percona1  |
 +|       2 | ssl       |
 ++---------+-----------+
 +2 rows in set (0.00 sec)
 +</code>
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki