boompty boomp

In http://blog.defunct.ca/2011/07/22/moving-nova-compute-to-a-separate-instance/, I was able to successfully move nova-compute to a separate instance. The only problem here is that the nova-compute instance used nova-network running on the controller, which introduced a single point of failure in our environment. If the controller dropped offline, the gateway for virtual machines running on the compute node would be inaccessible, meaning instances would not be able to access the outside world until the controller came back online.

Fortunately, some improvements have been made to Nova as outlined in http://unchainyourbrain.com/openstack/13-networking-in-nova. Essentially, we can now run a nova-network on each compute node, which forces the compute node to be the gateway for instances running on it. This means there’s no longer that dependency between the controller (or whatever runs nova-network) and virtual machines running on the compute node.

To move to this configuration, I had to run the following on the compute nodes:

# apt-get install nova-network

I then had to add the following configurations to the /etc/nova/nova.conf file on the compute nodes:

--ec2_dmz_host=192.168.0.1
--multi_host

Specifying –ec2_dmz_host=192.168.0.1 causes this iptables rule to get added:

Chain nova-network-PREROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DNAT       tcp  --  any    any     anywhere             169.254.169.254     tcp dpt:www to:192.168.0.1:8773

… and this allows cloud-init on the Ubuntu instances to grab whatever it is they’re grabbing from the EC2 API running on the controller. When the Ubuntu instances boot but can’t hit the EC2 API (I have 192.168.0.1 assigned to my controller, which runs the EC2 API), cloud-init seems to spin forever and the instances never really seems to boot. If you uninstall cloud-init, the instances will boot, but configuration does not appear to be complete (ie. missing ssh keys in /etc/ssh/). I tried using my controller’s public IP or the controller’s 10.176.65.54 address, but neither seemed to work. The latter is understandable as the instance will not be able to hit 10.176.65.54 since it’s not attached to that network, but it was my understanding that it should be able to hit the external IP.

Anyway, I also removed this from /etc/nova/nova.conf on the compute nodes as we no longer have to route through the controller:

--routing_source_ip=x.x.x.x

For good measure:

# /etc/init.d/nova-compute restart
# /etc/init.d/nova-network restart

Finally, I deleted my 192.168.0.0/24 on the controller and re-created it:

nova-manage network create --fixed_range_v4=192.168.0.0/24 --num_networks=1 --network_size=256 --multi_host=T --label=test

The key above is specifying the –multi_host=T.

This was more or less it. Now when an instance is first started on a compute node, the compute node itself gets an IP assigned from the network above and that IP gets assigned to the bridge br100. The instances on the host are then configured to use that IP as their gateway and traffic no longer gets routed through the controller.

One thing I noticed while working on this configuration was that my previous VPN connection didn’t permit multiple clients. As such, I had to move my VPN server/clients to use tls-server and tls-client, which required a bit more work (see this for more info).

My openvpn.server file:

mode server
tls-server
dev tap
ifconfig 192.168.0.1 255.255.0.0
cert /etc/openvpn/controller.crt
key /etc/openvpn/controller.key
dh /usr/share/doc/openvpn/examples/easy-rsa/2.0/keys/dh1024.pem
ca /usr/share/doc/openvpn/examples/easy-rsa/2.0/keys/ca.crt
daemon

… and openvpn.client for compute1:

tls-client
remote 10.176.65.54
dev tap
cert /etc/openvpn/compute1.crt
key /etc/openvpn/compute1.key
ca /etc/openvpn/ca.crt
daemon
keepalive 10 60
up /etc/openvpn/openvpn.up
up-restart
script-security 2

The /etc/openvpn/openvpn.up file contains:

#!/bin/bash
 
/sbin/ifconfig tap0 0.0.0.0 up
/usr/sbin/brctl addif br100 tap0
echo 0

Unlike our original configuration, br100 is IPd automatically by nova-network, so we no longer need to set an IP when openvpn starts on the clients. However, if the controller node (which subsequently runs the openvpn server) restarts, our clients cannot ping the 192.168.0.1 address even after the server comes back online. By adding the keepalive and up/up-restart entries to the openvpn.client file, we can force openvpn to get HUPd if the connection drops (or the server reboots).

There’s still a bit of magic happening here, but hopefully I’ve captured enough of this configuration to reconstruct this setup if necessary.

… turns out the image was missing /etc/init/ttyS0.conf:

 
# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.
 
start on stopped rc or RUNLEVEL=[2345]
stop on runlevel [!2345]
 
respawn
exec /sbin/getty -8 38400 ttyS0 vt102

Once I dropped that in there, I was able to console into the image.

Guessing most people are aware of this, but documenting for my own knowledge.

root@nova-cc:~# python
Python 2.7.1+ (r271:86832, Apr 11 2011, 18:13:53)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from novaclient.v1_0 import client
>>> client = client.Client(USERNAME, API_KEY,PROJECT_ID [, AUTH_URL])
>>> client.servers.list()
[<Server: Server 40>, <Server: Server 41>, <Server: Server 42>, <Server: Server 43>, <Server: Server 44>, <Server: Server 45>, <Server: Server 46>]
>>> for server in client.servers.list():
...     server.delete()
... 
>>> client.servers.list()
[]
>>>

The latest version of openstack-dashboard requires Keystone, and as I understand it this isn’t supported by the version of Nova I’m running (2011.3~d2-0ubuntu0~ppa1~natty1). Fortunately, I found this, which outlines how to use an older version of openstack-dashboard that does work without Keystone.

# apt-get update
# apt-get install bzr
# cd /root
# bzr branch lp:openstack-dashboard
# cd openstack-dashboard/
# bzr revert -r 46
# cd local
# cp -a local_settings.py.example local_settings.py

You now need to configure local_settings.py with correct values for NOVA_DEFAULT_ENDPOINT, NOVA_DEFAULT_REGION, NOVA_ACCESS_KEY, NOVA_SECRET_KEY, NOVA_ADMIN_USER, NOVA_PROJECT. Most of these values can be ripped out of novarc on your cloud controller.

Continue on with the installation:

# apt-get install -y python-setuptools
# easy_install virtualenv
# python tools/install_venv.py
# tools/with_venv.sh dashboard/manage.py syncdb

When you run “dashboard/manage.py syncdb”, it’ll prompt you asking if you want to create a Django superuser (since none exist at this point). I answered yes, entering a username that matched the value of NOVA_ADMIN_USER. I initially tried creating a Django user with a different username, and upon logging into openstack-dashboard I failed to see my Nova project.

Lastly, go ahead and start up the server:

# tools/with_venv.sh dashboard/manage.py runserver 0.0.0.0:8000

At this point, you should be able to access your dashboard on http://x.x.x.x:8000, replacing x.x.x.x with your openstack-dashboard server's IP.

If you run into any issues, refer to this, which contains valid information for this particular version of the dashboard.

Again, running with this old version of the dashboard isn't ideal, and you certainly don't want to run it as root, but hopefully this will point you in the right direction if you struggle to get the latest version to work with Nova. My next task will to be to get a version of the dashboard and Nova which work together installed and operational.

I want to quickly document how I accomplished this. Again, I used virtual machines (running Ubuntu Natty), but used public cloud server instances rather than private virtual machines.

First things first. Here’s the eth1 (private network) addresses assigned to my cloud servers:

nova-cc (our Nova cloud controller node):

eth1: 10.176.65.54

nova-compute (our Nova compute note, which will run our instances (QEMU or UML):

eth1: 10.176.95.220

Similar to previous posts, I went ahead and used 192.168.0.0/16 for Nova network as I didn’t have public IPs, nor did I want to interfere with the 10.176.64.0/18 network which is already used by this cloud provider.

On nova-cc, we need to install mysqld. Nova defaults to using SQLite, which works great when everything is running off a single instance. However, now that we’ve got another instance that needs to talk to nova-cc, we need a SQL server that it can connect to.

# apt-get update
# apt-get install mysql-server

Once installed, edit /etc/mysql/my.cnf and change this from:

bind-address            = 127.0.0.1

to:

bind-address            = 10.176.65.54

Finally, restart mysqld:

# service mysql restart

Now hop into the mysql shell and create a database and user/password to connect with:

# mysql -u root
mysql> CREATE DATABASE nova;
mysql> GRANT ALL PRIVILEGES ON nova.* TO nova@10.176.65.54 IDENTIFIED BY 'somepasshere';
mysql> GRANT ALL PRIVILEGES ON nova.* TO nova@10.176.95.220 IDENTIFIED BY 'somepasshere';

On nova-compute, the only nova-related package you really need is nova-compute:

# apt-get -y install python-software-properties
# add-apt-repository ppa:nova-core/milestone
# apt-get update
# apt-get install nova-compute

On both nova-cc and nova-compute:

# cat >> /etc/nova/nova.conf << "EOF"
--sql_connection=mysql://somepasshere@10.176.65.54/nova
--image_service=nova.image.glance.GlanceImageService
--glance_api_servers=10.176.65.54:9292
--rabbit_host=10.176.65.54
EOF
# for SERVICE in `ls -1 /etc/init.d/nova*`; do service $SERVICE restart; done

Now what we need to do is create our 192.68.0.0/16 network. We’ll use OpenVPN to do this, and we’ll use the eth1 private network (10.176.64.0/18) to do this this. Again, the idea here is to have a completely separate network which won’t interfere with what’s already out there.

On both:

# apt-get install openvpn

On nova-cc:

# cd /etc/openvpn
# openvpn --genkey --secret openvpn.key
# scp openvpn.key root@10.176.95.220:/etc/openvpn
# cat > /etc/openvpn/openvpn.server << "EOF"
dev tap
ifconfig 192.168.0.1 255.255.255.0
secret /etc/openvpn/openvpn.key
daemon
EOF
# cat > /etc/network/if-pre-up.d/00openvpn << "EOF"
#!/bin/bash
 
/usr/sbin/openvpn --config /etc/openvpn/openvpn.server
 
exit 0
EOF
# chmod 755 /etc/network/if-pre-up.d/00openvpn

On nova-compute:

# cat > /etc/openvpn/openvpn.client << "EOF"
remote 10.176.65.54
dev tap
ifconfig 192.168.0.3 255.255.255.0
secret /etc/openvpn/openvpn.key
daemon
EOF
# cat > /etc/network/if-pre-up.d/00openvpn << "EOF"
#!/bin/bash
 
/usr/sbin/openvpn --config /etc/openvpn/openvpn.client
 
/usr/sbin/brctl addbr br100
/usr/sbin/brctl addif br100 tap0
 
/sbin/ifconfig tap0 0.0.0.0
/sbin/ifconfig br100 192.168.0.3
 
exit 0
EOF
# chmod 755 /etc/network/if-pre-up.d/00openvpn

Finally, on both nodes:

# echo "--flat_interface=tap0" >> /etc/nova/nova/.conf

This causes nova-network to bridge into tap0.

Let’s recap what we’ve done. On nova-cc, we’re configuring OpenVPN to act as a server. We’re bringing tap0 up with IP 192.168.0.1/24 and the /etc/network/if-pre-up.d/00openvpn script ensures that the VPN server is started on on boot (specifically, before the other network devices are brought up). On nova-compute, we configure OpenVPN as a client, and the /etc/network/if-pre-up.d/00openvpn script creates a bridge (br100), adds the tap0 interface to it, and then brings 192.168.0.3 up on br100. If I recall correctly, the tap0 device doesn’t appear to be “up” until we ifconfig it, which is why we just set it to 0.0.0.0. Don’t quote me on this though, as I can’t quite remember.

I know very little about bridging, but essentially a bridge “connects two or more different physical ethernets together to form one large (logical) ethernet” (taken from /usr/share/doc/bridge-utils/HOWTO), and this is precisely what we have done here. We bridge the virtual interfaces for running instances (ie. vnet0) with tap0 (our VPN connection), which means that nova-cc can speak to instances running on nova-compute, and vice-versa. This is also essential as dnsmasq (our DHCP server) runs on nova-cc (spawned by nova-network), and without this bridging in place our instances would not be able to have their networking configured automatically on boot by the DHCP server.

Also, the reason why we don’t have to explicitly configure br100 on nova-cc is because that runs nova-network, which handles the bridging automatically. The only thing we did need to do on the nova-cc side is instruct nova-network on which device to bridge into (–flat_interface=tap0). The last thing I’ll say here is that OpenVPN used device tun0 by default, but we have to use tap0 (a virtual Ethernet adapter) as brctl creates Ethernet bridges, and a tun device is a “virtual point-to-point” link (see this for a tad more info).

Go ahead and reboot each instance, one at a time, to ensure that everything comes up as expected.

Once back up, on nova-cc:

# mysql -u root nova
mysql> SELECT * FROM fixed_ips WHERE id=4;
+---------------------+---------------------+------------+---------+----+-------------+------------+-------------+-----------+--------+----------+
| created_at          | updated_at          | deleted_at | deleted | id | address     | network_id | instance_id | allocated | leased | reserved |
+---------------------+---------------------+------------+---------+----+-------------+------------+-------------+-----------+--------+----------+
| 2011-07-22 16:21:35 | 2011-07-22 20:48:26 | NULL       |       0 |  4 | 192.168.0.3 |          1 |        NULL |         0 |      0 |        0 |
+---------------------+---------------------+------------+---------+----+-------------+------------+-------------+-----------+--------+----------+
1 row in set (0.00 sec)
 
mysql> UPDATE fixed_ips SET reserved=1 WHERE id=4 LIMIT 1;

What we’re doing here is “reserving” 192.168.0.3 for the other end of the VPN link on nova-compute. 192.168.0.2 is already reserved, but I’m not sure if nova uses this or will use it for something at some point. As such, just play it safe and reserve another available IP.

In theory, if you now launch an instance on nova-cc, it should build on nova-compute and the IP assigned should be accessible via nova-cc. The instance on nova-compute will have a gateway of 192.168.0.1 (which is physically on nova-cc), which means that all traffic in and out of the instance will travel through nova-cc. This also means that if nova-cc goes down, instances will not be able to communicate with the outside world (or potentially each other, though I’ve not tested myself).

That should be able it. I’ve probably missed a few things, but the general gist should be here. Also, I’m aware that there are no security best-practices implemented here, but the idea is to just get everything up and running as a proof of concept, and fine-tune later.

While creating UML instances on nova, I noticed I wasn’t able to ssh into my instances using the keypair I previously created. Looking at the logs on the nova-compute node, I saw:

2011-07-13 21:15:22,256 INFO nova.virt.libvirt_conn [-] instance instance-0000003d: injecting key into image 3
2011-07-13 21:15:22,256 DEBUG nova.utils [-] Running cmd (subprocess): sudo losetup --find --show /var/lib/nova/instances/instance-0000003d/disk from (pid=838) execute /usr/lib/pymodules/python2.7/nova/utils.py:143
2011-07-13 21:15:22,424 DEBUG nova.utils [-] Running cmd (subprocess): sudo kpartx -a /dev/loop0 from (pid=838) execute /usr/lib/pymodules/python2.7/nova/utils.py:143
2011-07-13 21:15:22,509 DEBUG nova.utils [-] Running cmd (subprocess): sudo kpartx -d /dev/loop0 from (pid=838) execute /usr/lib/pymodules/python2.7/nova/utils.py:143
2011-07-13 21:15:22,563 DEBUG nova.utils [-] Running cmd (subprocess): sudo losetup --detach /dev/loop0 from (pid=838) execute /usr/lib/pymodules/python2.7/nova/utils.py:143
2011-07-13 21:15:22,604 WARNING nova.virt.libvirt_conn [-] instance instance-0000003d: ignoring error injecting data into image 3 (Mapped device was not found (we can only inject raw disk images): /dev/mapper/loop0p1)

I tried running the kpartx commands above, but they didn’t return anything. This was because the image I was using had no partition table.

To fix, I effectively created a new image and copied data from the original one.

To begin, create a spare file (see this for more info):

# cd /root
# dd if=/dev/zero of=CentOS5.6-AMD64-new-root_fs bs=1 count=0 seek=1024M

Now, create a partition to span the entire disk (replace /dev/loop0 with whatever losetup returns):

# losetup --show --find CentOS5.6-AMD64-new-root_fs
/dev/loop0
# fdisk /dev/loop0

Now, use kpartx to make the partition visible to the host, and create a filesystem on that partition:

# parted
# kpartx -a /dev/loop0
# mke2fs -j /dev/mapper/loop0p1

Mount the original image and copy data over:

# losetup --show --find CentOS5.6-AMD64-root_fs 
/dev/loop1
# mkdir /mnt/loop{0,1}
# mount /dev/mapper/loop0p1 /mnt/loop0
# mount /dev/loop1 /mnt/loop1
# cd /mnt/loop1
# rsync -a . /mnt/loop0

Update the fstab on the new image (this is necessary as the partition layout has now changed):

# cd /mnt/loop0/etc
# sed -i 's/ubda/ubda1/g' fstab
# cd /
# umount /mnt/loop{0,1}
# kpartx -d /dev/loop0
# losetup -d /dev/loop{0,1}

Modify /etc/nova/libvirt.xml.template, changing this line from:

<root>/dev/ubda</root>

to:

<root>/dev/ubda1</root>

That should be about it.

As mentioned in http://blog.defunct.ca/?p=411, I’m running OpenStack on a XenServer virtual machine and need to be able to use something like UML to run instances from within the VM. I had to hack a number of things in order to get this to work.

First things first. Let’s get install user-mode-linux:

# apt-get update
# apt-get install user-mode-linux

Now, grab the CentOS 5.6 x86_64 image from http://fs.devloop.org.uk/ (we’re using a 64-bit XenServer VM):

# cd /root
# wget http://fs.devloop.org.uk/filesystems/CentOS-5.6/CentOS5.6-AMD64-root_fs.bz2
# bunzip2 CentOS5.6-AMD64-root_fs.bz2

There is an image on http://wiki.openstack.org/Nova/UML, however I couldn’t get this image to boot properly. As such, I opted for the CentOS 5.6 image above.

A few things within the image needed adjusting, so I:

# mkdir /mnt/image
# losetup --find --show CentOS5.6-AMD64-root_fs
/dev/loop0
# mount /dev/loop0 /mnt/image
# cp -a /usr/lib/uml/modules/2.6.35.1/ /mnt/image/lib/modules/
# chroot /mnt/image
# sed -i 's@LABEL=ROOT@/dev/ubda@g' /etc/fstab
# chkconfig network on
# exit
# umount /mnt/image
# losetup -d /dev/loop0

Note that /usr/lib/uml/modules/2.6.35.1/ is provided by the user-mode-linux package on Ubuntu 10.10, so adjust accordingly to what you’re running. Also, UML seems to use device /dev/ubda, so we modify /etc/fstab w/ that.

Now we can bundle up the image:

# cd /root
# euca-bundle-image -i CentOS5.6-AMD64-root_fs
# euca-upload-bundle -b uml-image-bucket -m /tmp/CentOS5.6-AMD64-root_fs.manifest.xml
# euca-register uml-image-bucket/CentOS5.6-AMD64-root_fs.manifest.xml

Once that’s done, we remove this from /etc/nova/nova.conf:

--libvirt_type=qemu

… and add this:

--libvirt_type=uml
--use_cow_images=false
--libvirt_xml_template=/etc/nova/libvirt.xml.template

Since I’ve specified –libvirt_xml_template in /etc/nova/nova.conf, we need to now create that file:

# cp -a /usr/share/pyshared/nova/virt/libvirt.xml.template /etc/nova/libvirt.xml.template

Now open up /etc/nova/libvirt.xml.template and remove the following:

#if $getVar('vncserver_host', False)
        <graphics type='vnc' port='-1' autoport='yes' keymap='en-us' listen='${vncserver_host}'/>
#end if

The reason for doing this is because it appears that we run into an issue similar to the one reported in here. Update: I’ve since created a bug report for this UML issue here.

Now we can restart the nova-compute service and create an instance:

# service nova-compute restart
# euca-run-instances ami-778c501e -k mykey -t m1.tiny

(replace ami-778c501e with your image name, which can be found by running euca-describe-images)

If your instance doesn’t go into a running state, have a look at the libvirt.xml file under /var/lib/nova/instances/####/ (replacing #### w/ your instance’s name, found by running euca-describe-instances), ensuring that there is no reference to the vnc stuff in there. If there is, then the template hasn’t been updated or isn’t being used correctly. Otherwise, your instance should be ssh-able, using the IP returned by euca-describe-instances.