boompty boomp

I want to quickly document how I accomplished this. Again, I used virtual machines (running Ubuntu Natty), but used public cloud server instances rather than private virtual machines.

First things first. Here’s the eth1 (private network) addresses assigned to my cloud servers:

nova-cc (our Nova cloud controller node):

eth1: 10.176.65.54

nova-compute (our Nova compute note, which will run our instances (QEMU or UML):

eth1: 10.176.95.220

Similar to previous posts, I went ahead and used 192.168.0.0/16 for Nova network as I didn’t have public IPs, nor did I want to interfere with the 10.176.64.0/18 network which is already used by this cloud provider.

On nova-cc, we need to install mysqld. Nova defaults to using SQLite, which works great when everything is running off a single instance. However, now that we’ve got another instance that needs to talk to nova-cc, we need a SQL server that it can connect to.

# apt-get update
# apt-get install mysql-server

Once installed, edit /etc/mysql/my.cnf and change this from:

bind-address            = 127.0.0.1

to:

bind-address            = 10.176.65.54

Finally, restart mysqld:

# service mysql restart

Now hop into the mysql shell and create a database and user/password to connect with:

# mysql -u root
mysql> CREATE DATABASE nova;
mysql> GRANT ALL PRIVILEGES ON nova.* TO nova@10.176.65.54 IDENTIFIED BY 'somepasshere';
mysql> GRANT ALL PRIVILEGES ON nova.* TO nova@10.176.95.220 IDENTIFIED BY 'somepasshere';

On nova-compute, the only nova-related package you really need is nova-compute:

# apt-get -y install python-software-properties
# add-apt-repository ppa:nova-core/milestone
# apt-get update
# apt-get install nova-compute

On both nova-cc and nova-compute:

# cat >> /etc/nova/nova.conf << "EOF"
--sql_connection=mysql://somepasshere@10.176.65.54/nova
--image_service=nova.image.glance.GlanceImageService
--glance_api_servers=10.176.65.54:9292
--rabbit_host=10.176.65.54
EOF
# for SERVICE in `ls -1 /etc/init.d/nova*`; do service $SERVICE restart; done

Now what we need to do is create our 192.68.0.0/16 network. We’ll use OpenVPN to do this, and we’ll use the eth1 private network (10.176.64.0/18) to do this this. Again, the idea here is to have a completely separate network which won’t interfere with what’s already out there.

On both:

# apt-get install openvpn

On nova-cc:

# cd /etc/openvpn
# openvpn --genkey --secret openvpn.key
# scp openvpn.key root@10.176.95.220:/etc/openvpn
# cat > /etc/openvpn/openvpn.server << "EOF"
dev tap
ifconfig 192.168.0.1 255.255.255.0
secret /etc/openvpn/openvpn.key
daemon
EOF
# cat > /etc/network/if-pre-up.d/00openvpn << "EOF"
#!/bin/bash
 
/usr/sbin/openvpn --config /etc/openvpn/openvpn.server
 
exit 0
EOF
# chmod 755 /etc/network/if-pre-up.d/00openvpn

On nova-compute:

# cat > /etc/openvpn/openvpn.client << "EOF"
remote 10.176.65.54
dev tap
ifconfig 192.168.0.3 255.255.255.0
secret /etc/openvpn/openvpn.key
daemon
EOF
# cat > /etc/network/if-pre-up.d/00openvpn << "EOF"
#!/bin/bash
 
/usr/sbin/openvpn --config /etc/openvpn/openvpn.client
 
/usr/sbin/brctl addbr br100
/usr/sbin/brctl addif br100 tap0
 
/sbin/ifconfig tap0 0.0.0.0
/sbin/ifconfig br100 192.168.0.3
 
exit 0
EOF
# chmod 755 /etc/network/if-pre-up.d/00openvpn

Finally, on both nodes:

# echo "--flat_interface=tap0" >> /etc/nova/nova/.conf

This causes nova-network to bridge into tap0.

Let’s recap what we’ve done. On nova-cc, we’re configuring OpenVPN to act as a server. We’re bringing tap0 up with IP 192.168.0.1/24 and the /etc/network/if-pre-up.d/00openvpn script ensures that the VPN server is started on on boot (specifically, before the other network devices are brought up). On nova-compute, we configure OpenVPN as a client, and the /etc/network/if-pre-up.d/00openvpn script creates a bridge (br100), adds the tap0 interface to it, and then brings 192.168.0.3 up on br100. If I recall correctly, the tap0 device doesn’t appear to be “up” until we ifconfig it, which is why we just set it to 0.0.0.0. Don’t quote me on this though, as I can’t quite remember.

I know very little about bridging, but essentially a bridge “connects two or more different physical ethernets together to form one large (logical) ethernet” (taken from /usr/share/doc/bridge-utils/HOWTO), and this is precisely what we have done here. We bridge the virtual interfaces for running instances (ie. vnet0) with tap0 (our VPN connection), which means that nova-cc can speak to instances running on nova-compute, and vice-versa. This is also essential as dnsmasq (our DHCP server) runs on nova-cc (spawned by nova-network), and without this bridging in place our instances would not be able to have their networking configured automatically on boot by the DHCP server.

Also, the reason why we don’t have to explicitly configure br100 on nova-cc is because that runs nova-network, which handles the bridging automatically. The only thing we did need to do on the nova-cc side is instruct nova-network on which device to bridge into (–flat_interface=tap0). The last thing I’ll say here is that OpenVPN used device tun0 by default, but we have to use tap0 (a virtual Ethernet adapter) as brctl creates Ethernet bridges, and a tun device is a “virtual point-to-point” link (see this for a tad more info).

Go ahead and reboot each instance, one at a time, to ensure that everything comes up as expected.

Once back up, on nova-cc:

# mysql -u root nova
mysql> SELECT * FROM fixed_ips WHERE id=4;
+---------------------+---------------------+------------+---------+----+-------------+------------+-------------+-----------+--------+----------+
| created_at          | updated_at          | deleted_at | deleted | id | address     | network_id | instance_id | allocated | leased | reserved |
+---------------------+---------------------+------------+---------+----+-------------+------------+-------------+-----------+--------+----------+
| 2011-07-22 16:21:35 | 2011-07-22 20:48:26 | NULL       |       0 |  4 | 192.168.0.3 |          1 |        NULL |         0 |      0 |        0 |
+---------------------+---------------------+------------+---------+----+-------------+------------+-------------+-----------+--------+----------+
1 row in set (0.00 sec)
 
mysql> UPDATE fixed_ips SET reserved=1 WHERE id=4 LIMIT 1;

What we’re doing here is “reserving” 192.168.0.3 for the other end of the VPN link on nova-compute. 192.168.0.2 is already reserved, but I’m not sure if nova uses this or will use it for something at some point. As such, just play it safe and reserve another available IP.

In theory, if you now launch an instance on nova-cc, it should build on nova-compute and the IP assigned should be accessible via nova-cc. The instance on nova-compute will have a gateway of 192.168.0.1 (which is physically on nova-cc), which means that all traffic in and out of the instance will travel through nova-cc. This also means that if nova-cc goes down, instances will not be able to communicate with the outside world (or potentially each other, though I’ve not tested myself).

That should be able it. I’ve probably missed a few things, but the general gist should be here. Also, I’m aware that there are no security best-practices implemented here, but the idea is to just get everything up and running as a proof of concept, and fine-tune later.

This entry was posted on Friday, July 22nd, 2011 at 10:50 pm and is filed under OpenStack. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.