OpenStack + Jclouds + Jenkins for corporate testing cloud

Disclaimer: This blog post is mostly fleshed out bullet points I've accumulated over the past weeks. Sorry for the slightly incoherent structure!

We've been looking to migrate from Eucalyptus to OpenStack. The main motivation for switching has been stability issues and general "bugginess" in Eucalyptus. Edit: Note that we're still on the eucalyptus version (2.0.1) that was bundled with ubuntu 10.04. No doubt many of the issues we've experienced are fixed in more recent versions of Eucalyptus.

I started looking at OpenStack early 2012, but it wasn't until summer 2012 that the Jenkins jclouds plugin was mature enough for us to be able to use it.

The strategy is to gradually move test jobs from eucalyptus to openstack, one image at a time.

Our current setup

Eucalyptus:
* 12 VM hosts, one cloud controller.
* All machines only use one NIC that's connected to the office network
* Managed by ec2-plugin running on jenkins

Openstack:
* 3 compute nodes, one controller
* FlatDHCPManager with multi_host off
* All nodes connected to internal physically disjunct network for cloud traffic on nic1
* All nodes connected to office network on nic2

We're currenly running all windows tests and some system tests on openstack. As we move more test jobs to openstack, we'll take machines from the eucalyptus cloud.

The good

The best thing openstack brought (apart from stability) was the ability to easily make changes on our images using snapshots. Fire up an instance, make your changes, snapshot and switch to the new snapshot in Jenkins. Works like a charm!

The bad

Multi host mode

multi_host mode not clearly documented. I never got this to work with vm:s getting internet access (only vms on the controller node could access internet). Enabling promisc mode on br100 on CC makes it work BUT instances running on the controller node would get dupe packets. Eventually I decided to not run nova-compute on the controller node, so this point might be moot. 

Delayed public IP

OpenStack can auto assign a public IP to VM:s, but the assignment occurs after boot. This means that when Jenkins provisions a node (via jclouds), it doesn't discover it's public IP. It would probably be possible to code around this in jclouds, but for now we've just added the controller node as a gateway for the private network on our jenkins host enabling jenkins to access the VM:s on their private IP.

Ephemeral storage on windows

It was a bit tricky to get ephemeral drives working correctly on windows. They were created and attached to kvm correctly, but were formatted with ext4. It turned out the docs were wrong, the correct way of setting up virt_mkfs options in nova.conf is:

virt_mkfs=windows=mkfs.ntfs --force --fast --label %(fs_label)s %(target)s
virt_mkfs=default=mkfs.ext4 -L %(fs_label)s -F %(target)s

The image also needs to have the os_type metadata set, i.e.
~$ nova image-meta c60f74c2-bc05-417a-b0a2-99456f6f4ef3 set os_type=windows

Default permissions on ephemeral drives in linux

Ephemeral drives are created with the following fstab entry:

/dev/vdb /mnt auto defaults,nobootwait,comment=cloudconfig 0 2

I haven't found a way of modifying for example the default permissions of the ephemeral drive (we would prefer it to have chmod 777).

Images are fetched on the public interface

The public interface is connected to a 100mbit network while the private network is 1gbps. I wanted to find a way to let the compute nodes fetch images from glance over the private network. The problem is that nova takes control over the private interface. Someone on IRC suggested that openvswitch could be used to create a fake interface, but I wasn't able to get it to work when I did some quick experimenting. Currently this isn't a major problem for us, since all images fit in the local image caches on the compute nodes.