INFN wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
======= Installation and Configuration of OpenStack Compute Node =======

Author:
   * Paolo E. Mazzon (DEI)
   * Matteo Menguzzato (DFA)
   * Gianpietro Sella (DISC)
===== Prerequisites =====
   * [[http://wiki.infn.it/progetti/cloud-areapd/ced-c/controller_and_network_node|Controller node install & setup]]

At least one node with:
   * Updated CentOS 7 (7.1)
   * Make sure that yum autoupdate is disabled
<code bash>
[root@cld-blu-11 ~]# grep ENA /etc/sysconfig/yum-autoupdate
# ENABLED
ENABLED="false"
</code> 
   * At least 20GB HD for operating system and OpenStack software and related log files 
   * Dedicated storage mounted on ''/var/lib/nova/instances'' where to store the instance images (particularly important to get live migration).
   * SELinux configured as "Disabled" (''/etc/selinux/config'')
   * EPEL 7
   * A MySQL (possibly a HA cluster) endpoint each OpenStack service can connect to (in this guide we're using our MySQL Percona cluster's IP 192.168.60.180)
   * A HAProxy/Keepalived cluster to use for load-balancing and Virtual IP (in this guide we're using the IP 192.168.60.180 for mgmt net and 90.147.143.10 for public net)
   * Disable NetworkManager
   * Installed CA INFN certificate on both nodes
   * add user "nova" to group "disk" and reboot node
<code bash>
[root@cld-blu-11 ~]# ll /etc/grid-security/chain.pem
-rw-r--r-- 1 root root 5607  9 apr 19.01 chain.pem
</code>
   * Installed and active libvirt
<code bash>
yum -y install libvirt
systemctl start libvirtd
systemctl enable libvirtd
</code>
   * Activated virtualization on CPU (can be toggled in the BIOS menu):
<code bash>
cat /proc/cpuinfo | grep vmx
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm dts tpr_shadow vnmi flexpriority

lsmod |grep kvm
kvm_intel              54285  12 
kvm                   332980  1 kvm_intel

lscpu |grep -i virtu
Virtualization:        VT-x

</code>
Note: ''kvm_intel'' can be substituted by ''kvm_amd'', and ''VT-x'' can be substituted by ''AMD-V''.
===== Naming conventions and networking assumptions =====
We assume that the compute node has the following network setup:
   * It has two network interface connected to two different networks: **management network** and **data network**
   * **Management network** is: ''192.168.60.0/24''
   * **Data network** is: ''192.168.61.0/24''
   * Node's IP are: ''192.168.60.XYZ'' and ''192.168.61.XYZ''

==== Further pre-requisite on data network interface ====
In the net-interface configuration script for data network (something like ''/etc/sysconfig/network-scripts/ifcfg-ethX'') put the following parameter:
<code bash>
MTU="9000"
</code>
===== IPTables configuration =====
Execute the following commands:
<code bash>
while read i
do
   firewall-cmd --add-port=${i}/tcp
   firewall-cmd --permanent --add-port=${i}/tcp
done << EOF
5900-5999
16509
49152-49261
EOF

# 5900-5999    VNC's TCP ports
# 16509        libvirtd's TCP ports
# 49152-49261  libvirtd's ephemeral ports

# permit ntpd's udp communications
firewall-cmd --add-port=123/udp
firewall-cmd --permanent --add-port=123/udp
firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 0 -p gre -j ACCEPT
</code>
===== Naming conventions and networking assumptions =====
We assume that the compute node has the following setup:
   * It has 2 network interfaces connected to two different networks: **management network**, **Data network**
   * **Management network** is: ''192.168.60.0/24''
   * **Data network** is: 192.168.61.0/24
   * The node is named: ''cld-blu-NN.cloud.pd.infn.it'' (''192.168.60.XYZ'', ''192.168.61.XYZ'')
   * In this guide the controller's VIP on the management network is needed: ''192.168.60.180''
   * In this guide the MySQL cluster's VIP on the management network is needed: ''192.168.60.180''
   * In this guide the controller's public IP is needed: ''90.147.143.10''
===== Install software =====
Install repo and Nova and Neutron's packages, and update ''iproute'' to support network namespaces:
<code bash>
yum -y install https://repos.fedorapeople.org/repos/openstack/EOL/openstack-icehouse/rdo-release-icehouse-4.noarch.rpm
yum -y install openstack-nova-compute openstack-utils openstack-neutron-openvswitch openstack-neutron-ml2 sysfsutils
yum -y update iproute
</code>


===== Preliminary networking setup =====
<code bash>
sed -i 's+^net\.ipv4\.conf\.default\.rp_filter+#net\.ipv4\.conf\.default\.rp_filter+' /etc/sysctl.conf
sed -i 's+^net\.ipv4\.conf\.all\.rp_filter+#net\.ipv4\.conf\.all\.rp_filter+' /etc/sysctl.conf
cat << EOF >> /etc/sysctl.conf
net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.default.rp_filter=0
net.bridge.bridge-nf-call-arptables=1
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
EOF
sysctl -p
systemctl restart network
</code>
===== Configure Nova =====
**nova.conf**
<code bash>
openstack-config --set /etc/nova/nova.conf database connection "mysql://nova:<NOVA_DB_PASS>@192.168.60.180:5306/nova"
openstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone
openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_host 192.168.60.180
openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_protocol http
openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_port 35357
openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_user nova
openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_tenant_name services
openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_password NOVA_PASS
openstack-config --set /etc/nova/nova.conf DEFAULT rpc_backend nova.openstack.common.rpc.impl_kombu
openstack-config --set /etc/nova/nova.conf DEFAULT rabbit_hosts 192.168.60.152:5672,192.168.60.153:5672
openstack-config --set /etc/nova/nova.conf libvirt live_migration_flag VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE
 
# Change the following IP with the actual IP of the current compute node on the management network
openstack-config --set /etc/nova/nova.conf DEFAULT my_ip 192.168.60.XYZ
openstack-config --set /etc/nova/nova.conf DEFAULT vnc_enabled True
# vncserver_listen MUST be 0.0.0.0 otherwise the live migration won't work correctly
# (http://docs.openstack.org/havana/config-reference/content/configuring-openstack-compute-basics.html#setting-flags-in-nova-conf-file)
openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_listen 0.0.0.0
 
# Change the following IP with the actual IP of the current compute node on the management network
openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_proxyclient_address 192.168.60.XYZ
openstack-config --set /etc/nova/nova.conf DEFAULT novncproxy_base_url http://90.147.143.10:6080/vnc_auto.html
openstack-config --set /etc/nova/nova.conf DEFAULT glance_host 192.168.60.180
openstack-config --set /etc/nova/nova.conf DEFAULT compute_driver nova.virt.libvirt.LibvirtDriver
openstack-config --set /etc/nova/nova.conf DEFAULT api_paste_config /etc/nova/api-paste.ini
openstack-config --set /etc/nova/nova.conf DEFAULT network_api_class nova.network.neutronv2.api.API
openstack-config --set /etc/nova/nova.conf DEFAULT neutron_url http://192.168.60.180:9696
openstack-config --set /etc/nova/nova.conf DEFAULT neutron_auth_strategy keystone
openstack-config --set /etc/nova/nova.conf DEFAULT neutron_admin_tenant_name services
openstack-config --set /etc/nova/nova.conf DEFAULT neutron_admin_username neutron
openstack-config --set /etc/nova/nova.conf DEFAULT neutron_admin_password NEUTRON_PASS
openstack-config --set /etc/nova/nova.conf DEFAULT neutron_admin_auth_url http://192.168.60.180:35357/v2.0
openstack-config --set /etc/nova/nova.conf DEFAULT linuxnet_interface_driver nova.network.linux_net.LinuxOVSInterfaceDriver
openstack-config --set /etc/nova/nova.conf DEFAULT security_group_api neutron
openstack-config --set /etc/nova/nova.conf DEFAULT firewall_driver nova.virt.firewall.NoopFirewallDriver
# the following 3 lines enable admin's password inject
openstack-config --set /etc/nova/nova.conf libvirt inject_password true
openstack-config --set /etc/nova/nova.conf libvirt inject_key true
openstack-config --set /etc/nova/nova.conf libvirt inject_partition -1
openstack-config --set /etc/nova/nova.conf DEFAULT neutron_metadata_proxy_shared_secret METADATA_PASS
openstack-config --set /etc/nova/nova.conf DEFAULT rabbit_ha_queues True
openstack-config --set /etc/nova/nova.conf DEFAULT cpu_allocation_ratio 4.0
openstack-config --set /etc/nova/nova.conf DEFAULT allow_resize_to_same_host true
 
 
# this is a temporary workaround untill we understand a problem of cpu not compatible when live-migrating VMs
openstack-config --set /etc/nova/nova.conf libvirt cpu_mode custom
openstack-config --set /etc/nova/nova.conf libvirt cpu_model kvm64

</code>
**api-paste.ini**
<code bash>
while read i
do
   openstack-config --set /etc/nova/api-paste.ini ${i}
done << EOF
filter:authtoken paste.filter_factory keystoneclient.middleware.auth_token:filter_factory
filter:authtoken auth_host 192.168.60.180
filter:authtoken auth_uri http://192.168.60.180:5000
filter:authtoken admin_tenant_name services
filter:authtoken admin_user nova
filter:authtoken admin_password NOVA_PASS
EOF
</code>


===== Configure LibVirt to support Live Migration =====
Turn OFF the ''libvirtd'' daemon :
<code bash>
systemctl stop libvirtd
</code>
Execute:
<code bash>
cat << EOF >> /etc/libvirt/libvirtd.conf
listen_tls = 0
listen_tcp = 1
auth_tcp = "none"
EOF
</code>
and
<code bash>
cat << EOF >> /etc/sysconfig/libvirtd
LIBVIRTD_ARGS="--listen"
EOF
</code>
Modify ''qemu.conf'':
<code bash>
cat << EOF >> /etc/libvirt/qemu.conf 
user="nova"
group="nova"
dynamic_ownership = 0
EOF
</code>
===== Configure Neutron's agents =====
As in the compute node the Neutron's L2 agent is running, some Neutron's configuration files need to be customized.

**neutron.conf**
<code bash>
openstack-config --set /etc/neutron/neutron.conf DEFAULT core_plugin ml2
openstack-config --set /etc/neutron/neutron.conf DEFAULT service_plugins router
openstack-config --set /etc/neutron/neutron.conf DEFAULT api_paste_config /etc/neutron/api-paste.ini
openstack-config --set /etc/neutron/neutron.conf DEFAULT rpc_backend neutron.openstack.common.rpc.impl_kombu
openstack-config --set /etc/neutron/neutron.conf DEFAULT rabbit_hosts 192.168.60.152:5672,192.168.60.153:5672
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_host 192.168.60.180
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_user neutron
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_password NEUTRON_PASS
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_url http://192.168.60.180:35357/v2.0
openstack-config --set /etc/neutron/neutron.conf DEFAULT auth_strategy keystone
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_tenant_name services
openstack-config --set /etc/neutron/neutron.conf agent root_helper "sudo neutron-rootwrap /etc/neutron/rootwrap.conf"
openstack-config --set /etc/neutron/neutron.conf database connection "mysql://neutron:<NEUTRON_DB_PWD>@192.168.60.180:4306/neutron"
openstack-config --set /etc/neutron/neutron.conf DEFAULT dhcp_agents_per_network 2
openstack-config --set /etc/neutron/neutron.conf DEFAULT dhcp_lease_duration 86400
openstack-config --set /etc/neutron/neutron.conf DEFAULT rabbit_ha_queues True
openstack-config --set /etc/neutron/neutron.conf DEFAULT agent_down_time 75 
openstack-config --set /etc/neutron/neutron.conf agent report_interval 30 
openstack-config --set /etc/neutron/neutron.conf filter:authtoken auth_port 35357
</code>

**ml2_conf.ini**

for local_ip parameter use the correct value
<code bash>
while read i
do
   openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ${i}
done << EOF
ml2 type_drivers gre
ml2 tenant_network_types gre
ml2 mechanism_drivers openvswitch
ml2_type_gre tunnel_id_ranges 1:1000
securitygroup firewall_driver neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
securitygroup enable_security_group True
ovs tenant_network_type gre
ovs tunnel_id_ranges 1:1000
ovs local_ip 192.168.61.XYZ
ovs enable_tunneling True
ovs integration_bridge br-int
ovs tunnel_bridge br-tun
EOF

ln -s /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugin.ini
</code>
(Neutron L2 agent's) **api-paste.ini**
<code bash>
openstack-config --set /etc/neutron/api-paste.ini filter:authtoken paste.filter_factory keystoneclient.middleware.auth_token:filter_factory
openstack-config --set /etc/neutron/api-paste.ini filter:authtoken auth_host 192.168.60.180
openstack-config --set /etc/neutron/api-paste.ini filter:authtoken admin_tenant_name services 
openstack-config --set /etc/neutron/api-paste.ini filter:authtoken admin_user neutron 
openstack-config --set /etc/neutron/api-paste.ini filter:authtoken admin_password NEUTRON_PASS
</code>
===== Bridge creation and start of the services =====
Open vSwitch start and bridge creation
<code bash>
systemctl start openvswitch
systemctl enable openvswitch
ovs-vsctl add-br br-int
</code>
L2 Agent configuration:
<code bash>
sed -i 's,plugins/openvswitch/ovs_neutron_plugin.ini,plugin.ini,g' /usr/lib/systemd/system/neutron-openvswitch-agent.service
systemctl daemon-reload
</code>
Mute neutron-openvswitch-agent eccessive logging:

The openvswitch service execs every two seconds the command ''ovs-ofctl dump-flows br-int table=22'' through ''sudo''. The execution of the command itself produces 43200 entries a day in ''/var/log/secure''. To mute the logging we add the ''!syslog'' directive to ''/etc/sudoers.d/neutron'' file:
<code bash>
sed -i 's/Defaults:neutron !requiretty$/Defaults:neutron !requiretty, !syslog/' /etc/sudoers.d/neutron
</code>
Start all the services:
<code bash>
systemctl start neutron-openvswitch-agent
systemctl enable neutron-openvswitch-agent
systemctl start libvirtd
systemctl start messagebus
systemctl start openstack-nova-compute
systemctl enable openstack-nova-compute
</code>
===== Check all =====
When done, log into the controller node, or wherever you've installed the Openstack CLI and copied the ''keystone_admin.sh'' into (which was created in the [[http://wiki.infn.it/progetti/cloud-areapd/keystone-glance_high_availability/openstack_ha/controller_node|controller node]] installation procedure). Execute the commands:
<code bash>
[root@cld-blu-03 ~]# neutron agent-list

+--------------------------------------+--------------------+-----------------------------+-------+----------------+
| id                                   | agent_type         | host                        | alive | admin_state_up |
+--------------------------------------+--------------------+-----------------------------+-------+----------------+
| 22cf4da6-3925-4a31-ba95-73e3332b1a36 | Open vSwitch agent | cld-blu-NN.cloud.pd.infn.it | :-)   | True           |
| 2d6aed8f-5a6d-46b1-8ce8-c07f4022d1d5 | DHCP agent         | cld-blu-03.cloud.pd.infn.it | :-)   | True           |
| 4d01a60a-b25c-4530-9860-c37490069e5e | Metadata agent     | cld-blu-03.cloud.pd.infn.it | :-)   | True           |
| 662f5529-675a-41e8-9490-4f0149edb6ce | Open vSwitch agent | cld-blu-03.cloud.pd.infn.it | :-)   | True           |
| c4c512b8-68cc-4deb-8451-d4c3d827809c | L3 agent           | cld-blu-03.cloud.pd.infn.it | :-)   | True           |
+--------------------------------------+--------------------+-----------------------------+-------+----------------+


[root@cld-blu-03 ~]# nova service-list
+------------------+-----------------------------+----------+---------+-------+----------------------------+-----------------+
| Binary           | Host                        | Zone     | Status  | State | Updated_at                 | Disabled Reason |
+------------------+-----------------------------+----------+---------+-------+----------------------------+-----------------+
| nova-consoleauth | cld-blu-03.cloud.pd.infn.it | internal | enabled | up    | 2015-03-27T10:26:58.000000 | -               |
| nova-conductor   | cld-blu-03.cloud.pd.infn.it | internal | enabled | up    | 2015-03-27T10:26:58.000000 | -               |
| nova-scheduler   | cld-blu-03.cloud.pd.infn.it | internal | enabled | up    | 2015-03-27T10:27:00.000000 | -               |
| nova-cert        | cld-blu-03.cloud.pd.infn.it | internal | enabled | up    | 2015-03-27T10:26:58.000000 | -               |
| nova-cert        | cld-blu-04.cloud.pd.infn.it | internal | enabled | up    | 2015-03-27T10:26:58.000000 | -               |
| nova-consoleauth | cld-blu-04.cloud.pd.infn.it | internal | enabled | up    | 2015-03-27T10:26:56.000000 | -               |
| nova-scheduler   | cld-blu-04.cloud.pd.infn.it | internal | enabled | up    | 2015-03-27T10:27:01.000000 | -               |
| nova-conductor   | cld-blu-04.cloud.pd.infn.it | internal | enabled | up    | 2015-03-27T10:27:04.000000 | -               |
| nova-compute     | cld-blu-NN.cloud.pd.infn.it | nova     | enabled | up    | 2015-03-27T10:27:02.000000 | -               |
+------------------+-----------------------------+----------+---------+-------+----------------------------+-----------------+
</code>

===== Add SSH passwordless access from Compute node to virtual instances =====
This is needed to allow nova to resize virtual instances. Execute the following commands:
<code bash>
usermod -s /bin/bash nova
mkdir -p -m 700 ~nova/.ssh
chown nova.nova ~nova/.ssh
cd ~nova/.ssh
scp cld-blu-03:/var/lib/nova/.ssh/* .
chown nova.nova * 
</code>

===== Optional: Configure Nova Compute for SSL =====
<code bash>
openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_host cloud.cedc.csia.unipd.it
openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_protocol https
openstack-config --set /etc/nova/nova.conf keystone_authtoken cafile /etc/grid-security/chain.pem
openstack-config --set /etc/nova/nova.conf DEFAULT neutron_ca_certificates_file /etc/grid-security/chain.pem
openstack-config --set /etc/nova/nova.conf DEFAULT cinder_ca_certificates_file /etc/grid-security/chain.pem
openstack-config --set /etc/nova/nova.conf DEFAULT neutron_admin_auth_url https://cloud.cedc.csia.unipd.it:35357/v2.0
openstack-config --set /etc/nova/nova.conf DEFAULT novncproxy_base_url https://cloud.cedc.csia.unipd.it:6080/vnc_auto.html
 
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_host cloud.cedc.csia.unipd.it
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_url https://cloud.cedc.csia.unipd.it:35357/v2.0
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_protocol https
openstack-config --set /etc/neutron/neutron.conf DEFAULT ssl_ca_file /etc/grid-security/chain.pem
openstack-config --set /etc/neutron/api-paste.ini filter:authtoken auth_host cloud.cedc.csia.unipd.it
 
openstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_host cloud.cedc.csia.unipd.it
openstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_protocol https
 
 
openstack-config --set /etc/nova/nova.conf DEFAULT glance_host cloud.cedc.csia.unipd.it
openstack-config --set /etc/nova/nova.conf DEFAULT glance_protocol https
openstack-config --set /etc/nova/nova.conf DEFAULT glance_api_servers https://cloud.cedc.csia.unipd.it:9292
openstack-config --set /etc/nova/nova.conf DEFAULT ssl_ca_file /etc/grid-security/chain.pem
#openstack-config --set /etc/nova/nova.conf ssl ca_file /etc/grid-security/chain.pem
openstack-config --set /etc/nova/nova.conf DEFAULT glance_api_insecure true
 
openstack-config --set /etc/nova/nova.conf DEFAULT neutron_url https://cloud.cedc.csia.unipd.it:9696
</code>
Restart L2 agent and Nova Compute
<code bash>
systemctl restart openstack-nova-compute
systemctl restart neutron-openvswitch-agent

</code>
=== Fix metadata agent ===
To address this [[https://bugs.launchpad.net/neutron/+bug/1263872|bug]], apply this [[https://review.openstack.org/#/c/79658/|patch]], or follow the instructions below:
<code bash>
curl -o agent.py https://raw.githubusercontent.com/CloudPadovana/SSL_Patches/master/agent.py
mv /usr/lib/python2.6/site-packages/neutron/agent/metadata/agent.py /usr/lib/python2.6/site-packages/neutron/agent/metadata/agent.py.bak
cp agent.py /usr/lib/python2.6/site-packages/neutron/agent/metadata/agent.py

service openstack-nova-compute restart
service neutron-openvswitch-agent restart

</code>

=== ===