======= Installation and Configuration of OpenStack Compute Node =======
Author:
* Paolo E. Mazzon (DEI)
* Matteo Menguzzato (DFA)
* Gianpietro Sella (DISC)
===== Prerequisites =====
* [[http://wiki.infn.it/progetti/cloud-areapd/ced-c/controller_and_network_node|Controller node install & setup]]
At least one node with:
* Updated CentOS 7 (7.1)
* Make sure that yum autoupdate is disabled
[root@cld-blu-11 ~]# grep ENA /etc/sysconfig/yum-autoupdate
# ENABLED
ENABLED="false"
* At least 20GB HD for operating system and OpenStack software and related log files
* Dedicated storage mounted on ''/var/lib/nova/instances'' where to store the instance images (particularly important to get live migration).
* SELinux configured as "Disabled" (''/etc/selinux/config'')
* EPEL 7
* A MySQL (possibly a HA cluster) endpoint each OpenStack service can connect to (in this guide we're using our MySQL Percona cluster's IP 192.168.60.180)
* A HAProxy/Keepalived cluster to use for load-balancing and Virtual IP (in this guide we're using the IP 192.168.60.180 for mgmt net and 90.147.143.10 for public net)
* Disable NetworkManager
* Installed CA INFN certificate on both nodes
* add user "nova" to group "disk" and reboot node
[root@cld-blu-11 ~]# ll /etc/grid-security/chain.pem
-rw-r--r-- 1 root root 5607 9 apr 19.01 chain.pem
* Installed and active libvirt
yum -y install libvirt
systemctl start libvirtd
systemctl enable libvirtd
* Activated virtualization on CPU (can be toggled in the BIOS menu):
cat /proc/cpuinfo | grep vmx
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm dts tpr_shadow vnmi flexpriority
lsmod |grep kvm
kvm_intel 54285 12
kvm 332980 1 kvm_intel
lscpu |grep -i virtu
Virtualization: VT-x
Note: ''kvm_intel'' can be substituted by ''kvm_amd'', and ''VT-x'' can be substituted by ''AMD-V''.
===== Naming conventions and networking assumptions =====
We assume that the compute node has the following network setup:
* It has two network interface connected to two different networks: **management network** and **data network**
* **Management network** is: ''192.168.60.0/24''
* **Data network** is: ''192.168.61.0/24''
* Node's IP are: ''192.168.60.XYZ'' and ''192.168.61.XYZ''
==== Further pre-requisite on data network interface ====
In the net-interface configuration script for data network (something like ''/etc/sysconfig/network-scripts/ifcfg-ethX'') put the following parameter:
MTU="9000"
===== IPTables configuration =====
Execute the following commands:
while read i
do
firewall-cmd --add-port=${i}/tcp
firewall-cmd --permanent --add-port=${i}/tcp
done << EOF
5900-5999
16509
49152-49261
EOF
# 5900-5999 VNC's TCP ports
# 16509 libvirtd's TCP ports
# 49152-49261 libvirtd's ephemeral ports
# permit ntpd's udp communications
firewall-cmd --add-port=123/udp
firewall-cmd --permanent --add-port=123/udp
firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 0 -p gre -j ACCEPT
===== Naming conventions and networking assumptions =====
We assume that the compute node has the following setup:
* It has 2 network interfaces connected to two different networks: **management network**, **Data network**
* **Management network** is: ''192.168.60.0/24''
* **Data network** is: 192.168.61.0/24
* The node is named: ''cld-blu-NN.cloud.pd.infn.it'' (''192.168.60.XYZ'', ''192.168.61.XYZ'')
* In this guide the controller's VIP on the management network is needed: ''192.168.60.180''
* In this guide the MySQL cluster's VIP on the management network is needed: ''192.168.60.180''
* In this guide the controller's public IP is needed: ''90.147.143.10''
===== Install software =====
Install repo and Nova and Neutron's packages, and update ''iproute'' to support network namespaces:
yum -y install https://repos.fedorapeople.org/repos/openstack/EOL/openstack-icehouse/rdo-release-icehouse-4.noarch.rpm
yum -y install openstack-nova-compute openstack-utils openstack-neutron-openvswitch openstack-neutron-ml2 sysfsutils
yum -y update iproute
===== Preliminary networking setup =====
sed -i 's+^net\.ipv4\.conf\.default\.rp_filter+#net\.ipv4\.conf\.default\.rp_filter+' /etc/sysctl.conf
sed -i 's+^net\.ipv4\.conf\.all\.rp_filter+#net\.ipv4\.conf\.all\.rp_filter+' /etc/sysctl.conf
cat << EOF >> /etc/sysctl.conf
net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.default.rp_filter=0
net.bridge.bridge-nf-call-arptables=1
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
EOF
sysctl -p
systemctl restart network
===== Configure Nova =====
**nova.conf**
openstack-config --set /etc/nova/nova.conf database connection "mysql://nova:@192.168.60.180:5306/nova"
openstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone
openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_host 192.168.60.180
openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_protocol http
openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_port 35357
openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_user nova
openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_tenant_name services
openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_password NOVA_PASS
openstack-config --set /etc/nova/nova.conf DEFAULT rpc_backend nova.openstack.common.rpc.impl_kombu
openstack-config --set /etc/nova/nova.conf DEFAULT rabbit_hosts 192.168.60.152:5672,192.168.60.153:5672
openstack-config --set /etc/nova/nova.conf libvirt live_migration_flag VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE
# Change the following IP with the actual IP of the current compute node on the management network
openstack-config --set /etc/nova/nova.conf DEFAULT my_ip 192.168.60.XYZ
openstack-config --set /etc/nova/nova.conf DEFAULT vnc_enabled True
# vncserver_listen MUST be 0.0.0.0 otherwise the live migration won't work correctly
# (http://docs.openstack.org/havana/config-reference/content/configuring-openstack-compute-basics.html#setting-flags-in-nova-conf-file)
openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_listen 0.0.0.0
# Change the following IP with the actual IP of the current compute node on the management network
openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_proxyclient_address 192.168.60.XYZ
openstack-config --set /etc/nova/nova.conf DEFAULT novncproxy_base_url http://90.147.143.10:6080/vnc_auto.html
openstack-config --set /etc/nova/nova.conf DEFAULT glance_host 192.168.60.180
openstack-config --set /etc/nova/nova.conf DEFAULT compute_driver nova.virt.libvirt.LibvirtDriver
openstack-config --set /etc/nova/nova.conf DEFAULT api_paste_config /etc/nova/api-paste.ini
openstack-config --set /etc/nova/nova.conf DEFAULT network_api_class nova.network.neutronv2.api.API
openstack-config --set /etc/nova/nova.conf DEFAULT neutron_url http://192.168.60.180:9696
openstack-config --set /etc/nova/nova.conf DEFAULT neutron_auth_strategy keystone
openstack-config --set /etc/nova/nova.conf DEFAULT neutron_admin_tenant_name services
openstack-config --set /etc/nova/nova.conf DEFAULT neutron_admin_username neutron
openstack-config --set /etc/nova/nova.conf DEFAULT neutron_admin_password NEUTRON_PASS
openstack-config --set /etc/nova/nova.conf DEFAULT neutron_admin_auth_url http://192.168.60.180:35357/v2.0
openstack-config --set /etc/nova/nova.conf DEFAULT linuxnet_interface_driver nova.network.linux_net.LinuxOVSInterfaceDriver
openstack-config --set /etc/nova/nova.conf DEFAULT security_group_api neutron
openstack-config --set /etc/nova/nova.conf DEFAULT firewall_driver nova.virt.firewall.NoopFirewallDriver
# the following 3 lines enable admin's password inject
openstack-config --set /etc/nova/nova.conf libvirt inject_password true
openstack-config --set /etc/nova/nova.conf libvirt inject_key true
openstack-config --set /etc/nova/nova.conf libvirt inject_partition -1
openstack-config --set /etc/nova/nova.conf DEFAULT neutron_metadata_proxy_shared_secret METADATA_PASS
openstack-config --set /etc/nova/nova.conf DEFAULT rabbit_ha_queues True
openstack-config --set /etc/nova/nova.conf DEFAULT cpu_allocation_ratio 4.0
openstack-config --set /etc/nova/nova.conf DEFAULT allow_resize_to_same_host true
# this is a temporary workaround untill we understand a problem of cpu not compatible when live-migrating VMs
openstack-config --set /etc/nova/nova.conf libvirt cpu_mode custom
openstack-config --set /etc/nova/nova.conf libvirt cpu_model kvm64
**api-paste.ini**
while read i
do
openstack-config --set /etc/nova/api-paste.ini ${i}
done << EOF
filter:authtoken paste.filter_factory keystoneclient.middleware.auth_token:filter_factory
filter:authtoken auth_host 192.168.60.180
filter:authtoken auth_uri http://192.168.60.180:5000
filter:authtoken admin_tenant_name services
filter:authtoken admin_user nova
filter:authtoken admin_password NOVA_PASS
EOF
===== Configure LibVirt to support Live Migration =====
Turn OFF the ''libvirtd'' daemon :
systemctl stop libvirtd
Execute:
cat << EOF >> /etc/libvirt/libvirtd.conf
listen_tls = 0
listen_tcp = 1
auth_tcp = "none"
EOF
and
cat << EOF >> /etc/sysconfig/libvirtd
LIBVIRTD_ARGS="--listen"
EOF
Modify ''qemu.conf'':
cat << EOF >> /etc/libvirt/qemu.conf
user="nova"
group="nova"
dynamic_ownership = 0
EOF
===== Configure Neutron's agents =====
As in the compute node the Neutron's L2 agent is running, some Neutron's configuration files need to be customized.
**neutron.conf**
openstack-config --set /etc/neutron/neutron.conf DEFAULT core_plugin ml2
openstack-config --set /etc/neutron/neutron.conf DEFAULT service_plugins router
openstack-config --set /etc/neutron/neutron.conf DEFAULT api_paste_config /etc/neutron/api-paste.ini
openstack-config --set /etc/neutron/neutron.conf DEFAULT rpc_backend neutron.openstack.common.rpc.impl_kombu
openstack-config --set /etc/neutron/neutron.conf DEFAULT rabbit_hosts 192.168.60.152:5672,192.168.60.153:5672
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_host 192.168.60.180
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_user neutron
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_password NEUTRON_PASS
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_url http://192.168.60.180:35357/v2.0
openstack-config --set /etc/neutron/neutron.conf DEFAULT auth_strategy keystone
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_tenant_name services
openstack-config --set /etc/neutron/neutron.conf agent root_helper "sudo neutron-rootwrap /etc/neutron/rootwrap.conf"
openstack-config --set /etc/neutron/neutron.conf database connection "mysql://neutron:@192.168.60.180:4306/neutron"
openstack-config --set /etc/neutron/neutron.conf DEFAULT dhcp_agents_per_network 2
openstack-config --set /etc/neutron/neutron.conf DEFAULT dhcp_lease_duration 86400
openstack-config --set /etc/neutron/neutron.conf DEFAULT rabbit_ha_queues True
openstack-config --set /etc/neutron/neutron.conf DEFAULT agent_down_time 75
openstack-config --set /etc/neutron/neutron.conf agent report_interval 30
openstack-config --set /etc/neutron/neutron.conf filter:authtoken auth_port 35357
**ml2_conf.ini**
for local_ip parameter use the correct value
while read i
do
openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ${i}
done << EOF
ml2 type_drivers gre
ml2 tenant_network_types gre
ml2 mechanism_drivers openvswitch
ml2_type_gre tunnel_id_ranges 1:1000
securitygroup firewall_driver neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
securitygroup enable_security_group True
ovs tenant_network_type gre
ovs tunnel_id_ranges 1:1000
ovs local_ip 192.168.61.XYZ
ovs enable_tunneling True
ovs integration_bridge br-int
ovs tunnel_bridge br-tun
EOF
ln -s /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugin.ini
(Neutron L2 agent's) **api-paste.ini**
openstack-config --set /etc/neutron/api-paste.ini filter:authtoken paste.filter_factory keystoneclient.middleware.auth_token:filter_factory
openstack-config --set /etc/neutron/api-paste.ini filter:authtoken auth_host 192.168.60.180
openstack-config --set /etc/neutron/api-paste.ini filter:authtoken admin_tenant_name services
openstack-config --set /etc/neutron/api-paste.ini filter:authtoken admin_user neutron
openstack-config --set /etc/neutron/api-paste.ini filter:authtoken admin_password NEUTRON_PASS
===== Bridge creation and start of the services =====
Open vSwitch start and bridge creation
systemctl start openvswitch
systemctl enable openvswitch
ovs-vsctl add-br br-int
L2 Agent configuration:
sed -i 's,plugins/openvswitch/ovs_neutron_plugin.ini,plugin.ini,g' /usr/lib/systemd/system/neutron-openvswitch-agent.service
systemctl daemon-reload
Mute neutron-openvswitch-agent eccessive logging:
The openvswitch service execs every two seconds the command ''ovs-ofctl dump-flows br-int table=22'' through ''sudo''. The execution of the command itself produces 43200 entries a day in ''/var/log/secure''. To mute the logging we add the ''!syslog'' directive to ''/etc/sudoers.d/neutron'' file:
sed -i 's/Defaults:neutron !requiretty$/Defaults:neutron !requiretty, !syslog/' /etc/sudoers.d/neutron
Start all the services:
systemctl start neutron-openvswitch-agent
systemctl enable neutron-openvswitch-agent
systemctl start libvirtd
systemctl start messagebus
systemctl start openstack-nova-compute
systemctl enable openstack-nova-compute
===== Check all =====
When done, log into the controller node, or wherever you've installed the Openstack CLI and copied the ''keystone_admin.sh'' into (which was created in the [[http://wiki.infn.it/progetti/cloud-areapd/keystone-glance_high_availability/openstack_ha/controller_node|controller node]] installation procedure). Execute the commands:
[root@cld-blu-03 ~]# neutron agent-list
+--------------------------------------+--------------------+-----------------------------+-------+----------------+
| id | agent_type | host | alive | admin_state_up |
+--------------------------------------+--------------------+-----------------------------+-------+----------------+
| 22cf4da6-3925-4a31-ba95-73e3332b1a36 | Open vSwitch agent | cld-blu-NN.cloud.pd.infn.it | :-) | True |
| 2d6aed8f-5a6d-46b1-8ce8-c07f4022d1d5 | DHCP agent | cld-blu-03.cloud.pd.infn.it | :-) | True |
| 4d01a60a-b25c-4530-9860-c37490069e5e | Metadata agent | cld-blu-03.cloud.pd.infn.it | :-) | True |
| 662f5529-675a-41e8-9490-4f0149edb6ce | Open vSwitch agent | cld-blu-03.cloud.pd.infn.it | :-) | True |
| c4c512b8-68cc-4deb-8451-d4c3d827809c | L3 agent | cld-blu-03.cloud.pd.infn.it | :-) | True |
+--------------------------------------+--------------------+-----------------------------+-------+----------------+
[root@cld-blu-03 ~]# nova service-list
+------------------+-----------------------------+----------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+-----------------------------+----------+---------+-------+----------------------------+-----------------+
| nova-consoleauth | cld-blu-03.cloud.pd.infn.it | internal | enabled | up | 2015-03-27T10:26:58.000000 | - |
| nova-conductor | cld-blu-03.cloud.pd.infn.it | internal | enabled | up | 2015-03-27T10:26:58.000000 | - |
| nova-scheduler | cld-blu-03.cloud.pd.infn.it | internal | enabled | up | 2015-03-27T10:27:00.000000 | - |
| nova-cert | cld-blu-03.cloud.pd.infn.it | internal | enabled | up | 2015-03-27T10:26:58.000000 | - |
| nova-cert | cld-blu-04.cloud.pd.infn.it | internal | enabled | up | 2015-03-27T10:26:58.000000 | - |
| nova-consoleauth | cld-blu-04.cloud.pd.infn.it | internal | enabled | up | 2015-03-27T10:26:56.000000 | - |
| nova-scheduler | cld-blu-04.cloud.pd.infn.it | internal | enabled | up | 2015-03-27T10:27:01.000000 | - |
| nova-conductor | cld-blu-04.cloud.pd.infn.it | internal | enabled | up | 2015-03-27T10:27:04.000000 | - |
| nova-compute | cld-blu-NN.cloud.pd.infn.it | nova | enabled | up | 2015-03-27T10:27:02.000000 | - |
+------------------+-----------------------------+----------+---------+-------+----------------------------+-----------------+
===== Add SSH passwordless access from Compute node to virtual instances =====
This is needed to allow nova to resize virtual instances. Execute the following commands:
usermod -s /bin/bash nova
mkdir -p -m 700 ~nova/.ssh
chown nova.nova ~nova/.ssh
cd ~nova/.ssh
scp cld-blu-03:/var/lib/nova/.ssh/* .
chown nova.nova *
===== Optional: Configure Nova Compute for SSL =====
openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_host cloud.cedc.csia.unipd.it
openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_protocol https
openstack-config --set /etc/nova/nova.conf keystone_authtoken cafile /etc/grid-security/chain.pem
openstack-config --set /etc/nova/nova.conf DEFAULT neutron_ca_certificates_file /etc/grid-security/chain.pem
openstack-config --set /etc/nova/nova.conf DEFAULT cinder_ca_certificates_file /etc/grid-security/chain.pem
openstack-config --set /etc/nova/nova.conf DEFAULT neutron_admin_auth_url https://cloud.cedc.csia.unipd.it:35357/v2.0
openstack-config --set /etc/nova/nova.conf DEFAULT novncproxy_base_url https://cloud.cedc.csia.unipd.it:6080/vnc_auto.html
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_host cloud.cedc.csia.unipd.it
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_url https://cloud.cedc.csia.unipd.it:35357/v2.0
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_protocol https
openstack-config --set /etc/neutron/neutron.conf DEFAULT ssl_ca_file /etc/grid-security/chain.pem
openstack-config --set /etc/neutron/api-paste.ini filter:authtoken auth_host cloud.cedc.csia.unipd.it
openstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_host cloud.cedc.csia.unipd.it
openstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_protocol https
openstack-config --set /etc/nova/nova.conf DEFAULT glance_host cloud.cedc.csia.unipd.it
openstack-config --set /etc/nova/nova.conf DEFAULT glance_protocol https
openstack-config --set /etc/nova/nova.conf DEFAULT glance_api_servers https://cloud.cedc.csia.unipd.it:9292
openstack-config --set /etc/nova/nova.conf DEFAULT ssl_ca_file /etc/grid-security/chain.pem
#openstack-config --set /etc/nova/nova.conf ssl ca_file /etc/grid-security/chain.pem
openstack-config --set /etc/nova/nova.conf DEFAULT glance_api_insecure true
openstack-config --set /etc/nova/nova.conf DEFAULT neutron_url https://cloud.cedc.csia.unipd.it:9696
Restart L2 agent and Nova Compute
systemctl restart openstack-nova-compute
systemctl restart neutron-openvswitch-agent
=== Fix metadata agent ===
To address this [[https://bugs.launchpad.net/neutron/+bug/1263872|bug]], apply this [[https://review.openstack.org/#/c/79658/|patch]], or follow the instructions below:
curl -o agent.py https://raw.githubusercontent.com/CloudPadovana/SSL_Patches/master/agent.py
mv /usr/lib/python2.6/site-packages/neutron/agent/metadata/agent.py /usr/lib/python2.6/site-packages/neutron/agent/metadata/agent.py.bak
cp agent.py /usr/lib/python2.6/site-packages/neutron/agent/metadata/agent.py
service openstack-nova-compute restart
service neutron-openvswitch-agent restart
=== ===