Author:

• Alvise Dorigo (INFN Padova)

• Controller node install & setup

Two nodes with:

• Updated SL6/CentOS6 (6.4 or 6.5)
• Make sure that yum autoupdate is disabled

root@controller-01 ~]# grep ENA /etc/sysconfig/yum-autoupdate
# ENABLED
ENABLED="false"

• At least 20GB HD for operating system and OpenStack software and related log files
• SELinux configured as "Disabled" (/etc/selinux/config)
• EPEL 6-8
• A MySQL (possibly a HA cluster) endpoint each OpenStack service can connect to (in this guide we use our MySQL Percona cluster's IP 192.168.60.10)
• A HAProxy/Keepalived cluster to use for load-balancing and Virtual IP (in this guide we use the IP 192.168.60.40 for mgmt net, 90.147.77.40 for public net)
• Installed CA INFN certificate on both nodes

[root@network-01 ~]# ll /etc/grid-security/certificates/INFN-CA-2006.pem 
-rw-r--r-- 1 root root 1257 Jun  5 19:05 /etc/grid-security/certificates/INFN-CA-2006.pem

We assume that the network nodes have the following network setup:

• They have 3 network interfaces connected to three different networks: management network, public network, data network
• Management network is: 192.168.60.0/24
• Public network (also called "external network") is: 90.147.77.0/24
• Data network is: 192.168.61.0/24
• First node is named: network-01.cloud.pd.infn.it (192.168.60.42), network-01.pd.infn.it (90.147.77.42)
• Second node is named: network-02.cloud.pd.infn.it (192.168.60.45), network-02.pd.infn.it (90.147.77.45)
• network-01 data network's IP is 192.168.61.42
• network-02 data network's IP is 192.168.61.45

In the net-interface configuration script for data network (something like /etc/sysconfig/network-scripts/ifcfg-XYZ) put the following parameter:

MTU="9000"

To make the Neutron agents highly available, just repeat this procedure on another network node, changing the value for the only relevant parameter local_ip (the private IP on the data network), and putting the correct value for the public network in the ifcfg-br-ex file, as shown below…

First install the YUM repo from RDO:

yum install -y http://rdo.fedorapeople.org/openstack-havana/rdo-release-havana.rpm

When the support to Havana is decomissioned, the repo changes. Then do the following:

yum -y install https://repos.fedorapeople.org/repos/openstack/EOL/openstack-havana/rdo-release-havana-9.noarch.rpm
sed -i 's+openstack/+openstack/EOL/+' /etc/yum.repos.d/rdo-release.repo

Then install Openstack software and update iproute in order to support network namespaces:

yum -y install openstack-neutron openvswitch openstack-neutron-openvswitch
yum -y update iproute

sed -i 's+^net\.ipv4.ip_forward+#net\.ipv4.ip_forward+' /etc/sysctl.conf
sed -i 's+^net\.ipv4\.conf\.default\.rp_filter+#net\.ipv4\.conf\.default\.rp_filter+' /etc/sysctl.conf
sed -i 's+^net\.ipv4\.conf\.all\.rp_filter+#net\.ipv4\.conf\.all\.rp_filter+' /etc/sysctl.conf
cat << EOF >> /etc/sysctl.conf
net.ipv4.ip_forward=1
net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.default.rp_filter=0
EOF
sysctl -p
service network restart

In this section we customize several configuration files related to Neutron's agents.

neutron.conf

# Let's choose the kind of authentication
openstack-config --set /etc/neutron/neutron.conf DEFAULT auth_strategy keystone
# Let's define the IP address and TCP port of the keystone service (which is running on the controller node)
# We'll use the Controller node's Virtual IP to exploit the HA configuration
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_host 192.168.60.40
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_port 35357
# Let's define the credentials used by the Neutron agents to authenticate to the Keystone service
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_tenant_name services
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_user neutron
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_password NEUTRON_PASS
# Let's use the RabbitMQ AMQP in HA mode
openstack-config --set /etc/neutron/neutron.conf DEFAULT rpc_backend neutron.openstack.common.rpc.impl_kombu
openstack-config --set /etc/neutron/neutron.conf DEFAULT rabbit_hosts 192.168.60.41:5672,192.168.60.44:5672
openstack-config --set /etc/neutron/neutron.conf agent root_helper "sudo neutron-rootwrap /etc/neutron/rootwrap.conf"
# Let's define the MySQL endpoint and authentication credentials
openstack-config --set /etc/neutron/neutron.conf database connection "mysql://neutron:<NEUTRON_DB_PWD>@192.168.60.10/neutron"
# Let's define the L2 plugin type (Open vSwitch or LinuxBridge; we're using the OVS)
openstack-config --set /etc/neutron/neutron.conf DEFAULT core_plugin neutron.plugins.openvswitch.ovs_neutron_plugin.OVSNeutronPluginV2
# The following parameter must contain the number of available dhcp agents, which is the number of network nodes. 2 in our case.
openstack-config --set /etc/neutron/neutron.conf DEFAULT dhcp_agents_per_network 2
openstack-config --set /etc/neutron/neutron.conf DEFAULT dhcp_lease_duration 86400
openstack-config --set /etc/neutron/neutron.conf DEFAULT rabbit_ha_queues True
openstack-config --set /etc/neutron/neutron.conf DEFAULT agent_down_time 75 
openstack-config --set /etc/neutron/neutron.conf agent report_interval 30 

api-paste.ini

openstack-config --set /etc/neutron/api-paste.ini filter:authtoken paste.filter_factory keystoneclient.middleware.auth_token:filter_factory
openstack-config --set /etc/neutron/api-paste.ini filter:authtoken auth_host 192.168.60.40
openstack-config --set /etc/neutron/api-paste.ini filter:authtoken auth_uri http://192.168.60.40:5000
openstack-config --set /etc/neutron/api-paste.ini filter:authtoken admin_tenant_name services
openstack-config --set /etc/neutron/api-paste.ini filter:authtoken admin_user neutron
openstack-config --set /etc/neutron/api-paste.ini filter:authtoken admin_password NEUTRON_PASS

l3-agent.ini

openstack-config --set /etc/neutron/l3_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.OVSInterfaceDriver
openstack-config --set /etc/neutron/l3_agent.ini DEFAULT use_namespaces True

dhcp_agent.ini

openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.OVSInterfaceDriver
openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT use_namespaces True
openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT dhcp_driver neutron.agent.linux.dhcp.Dnsmasq

metadata_agent.ini

openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_url http://192.168.60.40:5000/v2.0
openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_region regionOne
openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT admin_tenant_name services
openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT admin_user neutron
openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT admin_password NEUTRON_PASS
openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT nova_metadata_ip 192.168.60.40
openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT metadata_proxy_shared_secret METADATA_PASS

ovs_neutron_plugin.ini

openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs tenant_network_type gre
openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs tunnel_id_ranges 1:1000
# In the following line set local_ip to the IP address of the NIC connected to the DATA NETWORK
openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs local_ip 192.168.61.42
openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs enable_tunneling True
openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs integration_bridge br-int
openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs tunnel_bridge br-tun
openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini securitygroup firewall_driver neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
ln -s /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini /etc/neutron/plugin.ini

When using GRE the virtual instances can experience low network performances measured by iperf. This is because the ethernet packet is greatly used for GRE overhead information. To solve this problem you can increase to 9000 (this is a good value we've experienced) the MTU of the data network's switch, or do the following additional configuration:

openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT dnsmasq_config_file /etc/neutron/dnsmasq-neutron.conf

Create the dnsmasq's config file:

cat << EOF >> /etc/neutron/dnsmasq-neutron.conf
dhcp-option-force=26,1400
EOF

Turn the openvswitch ON:

service openvswitch start
chkconfig openvswitch on

Create the bridges:

ovs-vsctl add-br br-int
ovs-vsctl add-br br-ex

Let's assume the eth0 is the NIC attached to the external (public) network; move the public IP address from eth0 to br-ex and connect them:

cd /etc/sysconfig/network-scripts
mv ifcfg-eth0 eth0.orig
cat << EOF >> ifcfg-eth0
DEVICE=eth0
TYPE=OVSPort
DEVICETYPE=ovs
OVS_BRIDGE=br-ex
ONBOOT=yes
BOOTPROTO=none
PROMISC=yes
EOF
 
cat << EOF >> ifcfg-br-ex
DEVICE=br-ex
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
# change with your actual public IP address
IPADDR=90.147.77.42
NETMASK=255.255.255.0
ONBOOT=yes
EOF
 
service network restart
cd -

Turn the Neutron agents ON:

service neutron-dhcp-agent start
service neutron-l3-agent start
service neutron-metadata-agent start
service neutron-openvswitch-agent start

Enable the Neutron agents:

chkconfig neutron-dhcp-agent on
chkconfig neutron-l3-agent on
chkconfig neutron-metadata-agent on
chkconfig neutron-openvswitch-agent on

When you've done, you should be able to see all the agents running on all network nodes where you've applied this procedure. Execute the following command while logged into the controller node, or wherever you've installed the Openstack CLI and copied the keystone_admin.sh created in the guide for the controller node:

[root@controller-01 ~]# neutron agent-list
+--------------------------------------+--------------------+-----------------------------+-------+----------------+
| id                                   | agent_type         | host                        | alive | admin_state_up |
+--------------------------------------+--------------------+-----------------------------+-------+----------------+
| 188fe879-be8a-4390-b766-04e188e35c3c | L3 agent           | network-02.cloud.pd.infn.it | :-)   | True           |
| 42647a60-dbd0-4a85-942d-8fdbb0e2ae24 | Open vSwitch agent | network-01.cloud.pd.infn.it | :-)   | True           |
| cf6f7ec2-8700-498b-b62d-49d8b5616682 | DHCP agent         | network-02.cloud.pd.infn.it | :-)   | True           |
| dc249956-e81d-465c-b51f-cff0e1e04f05 | DHCP agent         | network-01.cloud.pd.infn.it | :-)   | True           |
| e196a6a2-8a3a-4bfe-b048-b50bee14761c | Open vSwitch agent | network-02.cloud.pd.infn.it | :-)   | True           |
| eb902101-8a16-43b5-87f8-b058530407f6 | L3 agent           | network-01.cloud.pd.infn.it | :-)   | True           |
+--------------------------------------+--------------------+-----------------------------+-------+----------------+

Configure files to use https and fully qualified hostname:

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_host cloud-areapd.pd.infn.it
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_protocol https
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_uri https://cloud-areapd.pd.infn.it:35357/
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_url https://cloud-areapd.pd.infn.it:35357/v2.0
openstack-config --set /etc/neutron/neutron.conf DEFAULT ssl_ca_file /etc/grid-security/certificates/INFN-CA-2006.pem
 
openstack-config --set /etc/neutron/api-paste.ini filter:authtoken auth_host cloud-areapd.pd.infn.it
openstack-config --set /etc/neutron/api-paste.ini filter:authtoken auth_uri https://cloud-areapd.pd.infn.it:5000
openstack-config --set /etc/neutron/api-paste.ini filter:authtoken auth_protocol https
 
openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_url https://cloud-areapd.pd.infn.it:5000/v2.0
openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_ca_cert /etc/grid-security/certificates/INFN-CA-2006.pem

Restart Neutron's agents:

service neutron-dhcp-agent restart
service neutron-l3-agent restart
service neutron-metadata-agent restart
service neutron-openvswitch-agent restart

To address this bug, apply this patch, or follow the instructions below:

curl -o agent.py https://raw.githubusercontent.com/CloudPadovana/SSL_Patches/master/agent.py
mv /usr/lib/python2.6/site-packages/neutron/agent/metadata/agent.py /usr/lib/python2.6/site-packages/neutron/agent/metadata/agent.py.bak
cp agent.py /usr/lib/python2.6/site-packages/neutron/agent/metadata/agent.py
 
service neutron-metadata-agent restart