Multi-node OpenStack RDO on RHEL and Hyper-V – Part 2

Here’s our second post in the RDO + Hyper-V series targeting Havana and automation.

We had lots of great feedbacks about the first post, along with many requests about how to troubleshoot network connectivity issues that arise quite often when you configure a complex environment like a multi-node OpenStack deployment, especially for the first time!

Although RDO is a great solution to automate an OpenStack deployment, it still requires quite a few manual steps, especially in a multi-node scenario. Any small configuration error in one of those steps can lead to hours of troubleshooting, so we definitely need an easy foolproof solution.

In this post we’ll detail a “simple” script available here that will perform all the configuration work for you.

Here’s a diagram of what we are going to obtain (click to enlarge):

 

Physical network layout

Network connectivity is very important, so let’s recap how the hosts need to be connected.

We’ll use 3 separate networks:

1. Management

This network will be used only for setting up the hosts and for HTTP and AMQP communication among them. It requires Internet access during the initial configuration. Guests must not be able to access this network under any circumstance.

2. Data (guest access)

Network used by the guests to communicate among each other (in the same Neutron network) and with the Network host for external routing.

Since we are going to use VLANs for network isolation in this example, make sure that your switches are properly configured (e.g. trunking mode).

3. External

Used by the Network host to communicate with external networks (e.g. Internet).

 

Hosts OS configuration

Now, we’ll need to get the hosts running. If you want to create a proof of concept (PoC), virtual machines can be absolutely enough as long as they support nested virtualization, while for a real production environment we suggest to use physical machines for all the nodes except optionally the controller. In this example we’ll use the “root” / “Passw0rd” credentials on all Linux hosts and “Administrator” / “Passw0rd” on Hyper-V.

Controller host

RAM

At least 1GB, 2-8GB or more if you plan to use Ceilometer on this node.

HDD

At least 10GB, recommended at least 100GB if you plan to store the Glance images here.

CPU

1-2 sockets, not particularly relevant in most circumstances.

Network

• eth0 connected to the Management network.

 

Network host

RAM

1GB

HDD

10GB or even less, this host has really no particular disk space requirements

CPU

1-2 sockets, not particularly relevant unless you use GRE / VXLAN tunnelling.

Network

• eth0 connected to the Management network.
• eth1 connected to the Data network.
• eth2 adapter connected to the External network.

 

KVM compute host (optional)

Multiple compute nodes can be added based on your requirements. We’ll use one in our example for the sake of simplicity.

RAM

A minimum of 4GB, Recommended? As much as possible. 🙂 This really depends on how many VMs you want to run. Sum the amount of memory you want to assign to the guests and add 1 GB for the host. Avoid memory overcommit if possible. 64, 128, 192 GB are quite common choices.

HDD

A minimum of 10GB. This is based on the local storage that you want to assign to the guests, I’d start with at least 100GB.

CPU

2 sockets, as many cores as possible, based on the number of guests. For example 2 esacore sockets with multithreading enabled provide 12 usable CPU threads to assign to the guests.

Network

• eth0 adapter connected to the Management network.
• eth1 adapter connected to the Data network.

Hyper-V compute host (optional)

The same consideration discussed for the KVM compute node apply to this case as well.

You can just perform a basic installation and at the end enable WinRM by running the following Powershell script that will download OpenSSL, configure a CA, generate a certificate to be used for HTTPS and configure WinRM for HTTPS usage with basic authentication: SetupWinRMAccess.ps1

 

Configure RDO

Now, let’s get the management IPs assigned to the hosts. In our examples we’ll use:

1. Controller: 192.168.200.1
2. Network: 192.168.200.2
3. KVM compute: 192.168.200.3
4. Hyper-V compute: 192.168.200.4

 

You can already start the script and let it run while we explain how it works afterwards. It runs on Linux, Mac OS X or Windows (using Bash).

If you are not deploying Hyper-V, you can simply omit the last two values of course.

Here’s a video showing what to expect (except the initial VM deployment part which is outside of the scope of this post):

[framed_video column=”full-width”]http://www.youtube.com/watch?v=1ncoEews1QA[/framed_video]

The script explained

You can open it directly on Github and follow the next explanations:

1. Inject the SSH key

This is performed for each Linux host to enable the automated configuration steps afterwards.

2. Set the host date and time

3. Rename the Hyper-V compute host (if present)

This uses a separate script which in turn is based on WSMan to call Powershell remotely (even from Linux!):

4. Configure Linux networking on all nodes

This uses a function called config_openstack_network_adapter to configure the adapter’s ifcfg-ethx file to be used by Open vSwitch.

5. Test if networking works

This is a very important step, as it validates the network configuration anticipating potential issues!
Some helper functions are used in this case as well.

6. Install Packstack on the controller

We’re also adding crudini to manage the OpenStack configuration files.

7. Generate a Packstack configuration file

8. Configure the Packstack answer file

Here’s the part relevant to Havana.

9. Deploy our SSH key on the controller

The key will be used by Packstack to automate the SSH connection with all the nodes.

10. Run Packstack!

This is obviously the most important step. Note the use of a helper function called run_ssh_cmd_with_retry to retry the command if it fails.
Transient errors in Packstack are quite common due to network connectivity issues, nothing to worry about.

11. Add additional firewall rules

Enable access from the Hyper-V host(s) and work around an existing Packstack issue in multinode environments.

12. Disable API rate limits

This is useful for PoCs and testing environments, but you most probably want to leave API rates in place in production.

13. Enable Neutron firewall

Fixes a Packstack glitch.

14. Set KVM as the libvirt type on the KVM compute node

When running on a VM, Packstack sets always the libvirt_type to QEMU, which is not what we want.

15. Configure Open vSwitch on the Network node

16. Reboot all the Linux nodes to load the new 2.6.32 Kernel with network namespaces support

17. Install OpenStack on Hyper-V

Configures remotely a Hyper-V external virtual switch and installs the OpenStack compute node with the Nova installer.

Here are some details from this script.

Create a Hyper-V external virtual switch using Powershell remotely.

Run the Nova installer remotely in unattended mode.

18. Make sure everything is running fine!

Check if all Nova services are running (including the optional KVM and Hyper-V compute nodes):

An finally check that all the Neutron agents are up and running: