OpenStack on Hyper-V – Icehouse 2014.1.3 – Best tested compute component?

Releasing stable components of a large cloud computing platform like OpenStack is not something that can be taken lightheartedly, there are simply too many variables and moving parts that need to be taken in consideration.

The OpenStack development cycle includes state of the art continuous integration testing including a large number of 3rd party CI testing infrastructures to make sure that any new code contribution won’t break the existing codebase.

The OpenStack on Hyper-V 3rd party CI is currently available for Nova and Neutron (with Cinder support in the works and more projects along the way), spinning up an OpenStack cloud with Hyper-V nodes for every single new code patchset to be tested, meaning hundreds of clouds deployed and dismissed per day. It’s hosted by Microsoft and maintained by a team composed by Microsoft and Cloudbase Solutions engineers.

This is a great achievement, especially when cnsidered in the whole OpenStack picture, where dozens of other testing infrastructures operate in a similar way while hundreds of developers tirelessly submit code to be reviewed. Thanks to this large scale joint effort, QA automation has surely been taken to a whole new level.
 

Where’s the catch?

There’s always a tradeoff between the desired workload and the available resources. In an ideal world, we would test every possible use case scenario, including all combinations of supported operating systems and component configurations. The result would simply require too many resources or execution times in the order of a few days. Developers and reviewers need to know if the code passed tests, so long test execution times are simply detrimental for the project. A look at the job queue shortly before a code freeze day will give a very clear idea of what we are talking about :-).

On the other side, stable releases require as much testing as possible, especially if you plan to sleep at night while your customers deploy your products in production environments.

To begin with, the time constraint that continuous integration testing requires disappear, since in OpenStack we have a release every month or so and this leads us to:

Putting the test scenarios together

We just need a matrix of operating systems and project specific options combinations that we want to test. The good news here are that the actual tests to be performed are the same ones used for continuous integration (Tempest), simply repeated for every scenario.

For the specific Hyper-V compute case, we need to test features that the upstream OpenStack CI infrastructure cannot test. Here’s a quick rundown list:

• Every supported OS version: Hyper-V 2008 R2, 2012, 2012 R2 and vNext.
• Live migration, which requires 2 compute servers per run
• VHD and VHDX images (fixed, dynamic)
• Copy on Write (CoW) and full clones 
• Various Neutron network configurations: VLAN, flat and soon Open vSwitch!
• Dynamic / fixed VM memory
• API versions (v1, v2)
• A lot more coming with the Kilo release: Hyper-V Generation 2 VMs, RemoteFX, etc

Downstream bug fixes and features

Another reason for performing additional tests is that “downstream” product releases, integrate the “upstream” projects (the ones available on the Launchpad project page and related git repositories) with critical bug fixes not yet merged upstream (time to land a patch is usually measured in weeks) and optionally new features backported from subsequent releases.

For example the OpenStack Hyper-V Icehouse 2014.1.3 release includes the following additions:

Nova

• Hyper-V: cleanup basevolumeutils
• Hyper-V: Skip logging out in-use targets
• Fixes spawn issue on Hyper-V
• Fixes Hyper-V dynamic memory issue with vNUMA
• Fixes differencing VHDX images issue on Hyper-V
• Fixes Hyper-V should log a clear error message
• Fixes HyperV VM Console Log
• Adds Hyper-V serial console log
• Adds Hyper-V Compute Driver soft reboot implementation
• Fixes Hyper-V driver WMI issue on 2008 R2
• Fixes Hyper-V boot from volume live migration
• Fixes Hyper-V volume discovery exception message
• Add differencing vhdx resize support in Hyper-V Driver
• Fixes Hyper-V volume mapping issue on reboot
• HyperV Driver – Fix to implement hypervisor-uptime

Neutron

• Fixes Hyper-V agent port disconnect issue
• Fixes Hyper-V 2008 R2 agent VLAN Settings issue
• Fixes Hyper-V agent stateful security group rules

Ceilometer

• No changes from upstream

Running all the relevant integration tests against the updated repositories provides an extremely important proof for our users that the quality standards are well respected.

Source code repositories:

• https://github.com/cloudbase/nova/tree/2014.1.3-cloudbase-release
• https://github.com/cloudbase/neutron/tree/2014.1.3-cloudbase-release

 

Packaging

Since we released the first Hyper-V installer for Folsom we had a set goals:

• Easy to deploy
• Automated configuration
• Unattended installation
• Include a dedicated Python environment
• Easy to automate with Puppet, Chef, SaltStack, etc
• Familiar for Windows users
• Familiar for DevOps
• Handle required OS configurations (e.g. create VMSwitches)
• No external requirements / downloads
• Atomic deployment

The result is the Hyper-V OpenStack MSI installer that keeps getting better with every release:

 

Sharing the test results

Starting with Icehouse 2014.1.3 we decided to publish the test results and the tools that we use to automate the tests execution:

Test results

http://www.cloudbase.it/openstack-hyperv-release-tests-results

Each release contains a subfolder for every test execution (Hyper-V 2012 R2 VHDX, Hyper-V 2012 VHD, etc), which in turn will contain the results in HTML format and every possible log, configuration file, list of applied Windows Update hot fixes, DevStack logs and so on.

Test tools

All the scripts that we are using are available here:

https://github.com/cloudbase/openstack-hyperv-release-tests

The main goal is to provide a set of tools that anybody can use efficiently with minimum hardware requirements and reproduce the same tests that we run (see for example the stack of Intel NUCs above).

Hosts:

• Linux host running Ubuntu 12.04 or 14.04
• One or more Hyper-V nodes

Install the relevant prerequisites on the Linux node.

Enable WinRM with HTTPS on the Hyper-V nodes.

Edit config.yaml, providing the desired Hyper-V node configurations and run:

The execution can be easily integrated with Jenkins or any other automation tool:

Run with custom parameters, for testing individual platforms:

We are definitely happy with the way in which Hyper-V support in OpenStack is growing. We are adding lots of new features and new developers keep on joining the ranks, so QA became an extremely important part of the whole equation. Our goal is to keep the process open so that anybody can review and contribute to our testing procedures for both the stable releases and the master branch testing executed on the Hyper-V CI infrastructure.