These installation instructions are basically the steps from http://graphite.wikidot.com/installation, with a few minor adjustments to work on Debian Squeeze.

First things first:

# cd /root
# apt-get update
# apt-get install bzr

Once bzr’s installed, we can:

# bzr branch lp:graphite

Install Whisper:

# cd graphite/whisper
# python setup.py install

At the time of writing, Whisper can be installed from apt-get on Squeeze, but the version doesn’t match what we’ve pulled Launchpad.

Install Carbon:

# cd ../carbon
# python setup.py install

Now we copy some sample configurations into place:

# cd /opt/graphite/conf
# cp carbon.conf.example carbon.conf
# cp storage-schemas.conf.example storage-schemas.conf

Graphite depends on a number of other packages, and I’ve made every attempt to grab stuff from stock apt repos rather than building (unnecessarily) from source. To see what’s missing, run:

cd /root/graphite
python check-dependencies.py

This should return something like this:

# python check-dependencies.py
[FATAL] Unable to import the 'cairo' module, do you have pycairo installed for python 2.6.6?
[FATAL] Unable to import the 'django' module, do you have Django installed for python 2.6.6?
[FATAL] Unable to import the 'tagging' module, do you have django-tagging installed for python 2.6.6?
[WARNING] Unable to import Interface from zope.interface.
Without it, you will be unable to run carbon on this server.
[WARNING] Unable to import the 'mod_python' module, do you have mod_python installed for python 2.6.6?
mod_python is one of the most common ways to run graphite-web under apache.
Without mod_python you will still be able to use the built in development server; which is not
recommended for production use.
wsgi or other approaches for production scale use are also possible without mod_python
[WARNING]
Unable to import the 'memcache' module, do you have python-memcached installed for python 2.6.6?
This feature is not required but greatly improves performance.
 
[WARNING]
Unable to import the 'ldap' module, do you have python-ldap installed for python 2.6.6?
Without python-ldap, you will not be able to use LDAP authentication in the graphite webapp.
 
[WARNING]
Unable to import the 'twisted' package, do you have Twisted installed for python 2.6.6?
Without Twisted, you cannot run carbon on this server.
[WARNING]
Unable to import the 'txamqp' module, this is required if you want to use AMQP.
Note that txamqp requires python 2.5 or greater.
3 necessary dependencies not met. Graphite will not function until these dependencies are fulfilled.
6 optional dependencies not met. Please consider the warning messages before proceeding.

Now, to get this stuff installed:

# apt-get install python-cairo
# apt-get install python-django-tagging
# apt-get install python-twisted
# apt-get install python-memcache
# apt-get install libapache2-mod-wsgi

I don’t know enough about mod_python (which we’re supposed to install), but the sample vhost configuration below refers to mod_wsgi, so I installed that instead.

Now we configure Apache and modify the sample vhost configuration file provided:

# rm /etc/apache2/sites-enabled/000-default
# cp -a examples/example-graphite-vhost.conf /etc/apache2/sites-enabled/graphite
# cp -a conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi
# sed -i s%"@DJANGO_ROOT@/contrib/admin/media/"%"/usr/share/pyshared/django/contrib/admin/media/"% /etc/apache2/sites-enabled/graphite
# sed -i 's%WSGISocketPrefix /etc/httpd/wsgi/%WSGISocketPrefix /var/run/apache2/wsgi%' /etc/apache2/sites-enabled/graphite

Once done, go ahead and restart Apache:

# /etc/init.d/apache2 reload

Finally:

# cd /opt/graphite/webapp/graphite
# python manage.py syncdb
# chown -R www-data:www-data /opt/graphite/storage/
# cd /opt/graphite/
# ./bin/carbon-cache.py start

If that’s all worked, you should be able to pump data into Graphite:

# echo "carbon.installation.test $RANDOM `date +%s`" | nc -w 1 localhost 2003

On the above, if I don’t pass -w 1 to nc, nc just sits there doing nothing.

Now, browsing your server’s http://x.x.x.x/ should load up the Graphite app and hopefully you can see a graph for the data you’ve sent in.

Fortunately, some improvements have been made to Nova as outlined in http://unchainyourbrain.com/openstack/13-networking-in-nova. Essentially, we can now run a nova-network on each compute node, which forces the compute node to be the gateway for instances running on it. This means there’s no longer that dependency between the controller (or whatever runs nova-network) and virtual machines running on the compute node.

To move to this configuration, I had to run the following on the compute nodes:

# apt-get install nova-network

I then had to add the following configurations to the /etc/nova/nova.conf file on the compute nodes:

--ec2_dmz_host=192.168.0.1
--multi_host

Specifying –ec2_dmz_host=192.168.0.1 causes this iptables rule to get added:

Chain nova-network-PREROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DNAT       tcp  --  any    any     anywhere             169.254.169.254     tcp dpt:www to:192.168.0.1:8773

… and this allows cloud-init on the Ubuntu instances to grab whatever it is they’re grabbing from the EC2 API running on the controller. When the Ubuntu instances boot but can’t hit the EC2 API (I have 192.168.0.1 assigned to my controller, which runs the EC2 API), cloud-init seems to spin forever and the instances never really seems to boot. If you uninstall cloud-init, the instances will boot, but configuration does not appear to be complete (ie. missing ssh keys in /etc/ssh/). I tried using my controller’s public IP or the controller’s 10.176.65.54 address, but neither seemed to work. The latter is understandable as the instance will not be able to hit 10.176.65.54 since it’s not attached to that network, but it was my understanding that it should be able to hit the external IP.

Anyway, I also removed this from /etc/nova/nova.conf on the compute nodes as we no longer have to route through the controller:

--routing_source_ip=x.x.x.x

For good measure:

# /etc/init.d/nova-compute restart
# /etc/init.d/nova-network restart

Finally, I deleted my 192.168.0.0/24 on the controller and re-created it:

nova-manage network create --fixed_range_v4=192.168.0.0/24 --num_networks=1 --network_size=256 --multi_host=T --label=test

The key above is specifying the –multi_host=T.

This was more or less it. Now when an instance is first started on a compute node, the compute node itself gets an IP assigned from the network above and that IP gets assigned to the bridge br100. The instances on the host are then configured to use that IP as their gateway and traffic no longer gets routed through the controller.

One thing I noticed while working on this configuration was that my previous VPN connection didn’t permit multiple clients. As such, I had to move my VPN server/clients to use tls-server and tls-client, which required a bit more work (see this for more info).

My openvpn.server file:

mode server
tls-server
dev tap
ifconfig 192.168.0.1 255.255.0.0
cert /etc/openvpn/controller.crt
key /etc/openvpn/controller.key
dh /usr/share/doc/openvpn/examples/easy-rsa/2.0/keys/dh1024.pem
ca /usr/share/doc/openvpn/examples/easy-rsa/2.0/keys/ca.crt
daemon

… and openvpn.client for compute1:

tls-client
remote 10.176.65.54
dev tap
cert /etc/openvpn/compute1.crt
key /etc/openvpn/compute1.key
ca /etc/openvpn/ca.crt
daemon
keepalive 10 60
up /etc/openvpn/openvpn.up
up-restart
script-security 2

The /etc/openvpn/openvpn.up file contains:

#!/bin/bash
 
/sbin/ifconfig tap0 0.0.0.0 up
/usr/sbin/brctl addif br100 tap0
echo 0

Unlike our original configuration, br100 is IPd automatically by nova-network, so we no longer need to set an IP when openvpn starts on the clients. However, if the controller node (which subsequently runs the openvpn server) restarts, our clients cannot ping the 192.168.0.1 address even after the server comes back online. By adding the keepalive and up/up-restart entries to the openvpn.client file, we can force openvpn to get HUPd if the connection drops (or the server reboots).

There’s still a bit of magic happening here, but hopefully I’ve captured enough of this configuration to reconstruct this setup if necessary.

 
# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.
 
start on stopped rc or RUNLEVEL=[2345]
stop on runlevel [!2345]
 
respawn
exec /sbin/getty -8 38400 ttyS0 vt102

Once I dropped that in there, I was able to console into the image.

# apt-get update
# apt-get install bzr
# cd /root
# bzr branch lp:openstack-dashboard
# cd openstack-dashboard/
# bzr revert -r 46
# cd local
# cp -a local_settings.py.example local_settings.py

You now need to configure local_settings.py with correct values for NOVA_DEFAULT_ENDPOINT, NOVA_DEFAULT_REGION, NOVA_ACCESS_KEY, NOVA_SECRET_KEY, NOVA_ADMIN_USER, NOVA_PROJECT. Most of these values can be ripped out of novarc on your cloud controller.

Continue on with the installation:

# apt-get install -y python-setuptools
# easy_install virtualenv
# python tools/install_venv.py
# tools/with_venv.sh dashboard/manage.py syncdb

When you run “dashboard/manage.py syncdb”, it’ll prompt you asking if you want to create a Django superuser (since none exist at this point). I answered yes, entering a username that matched the value of NOVA_ADMIN_USER. I initially tried creating a Django user with a different username, and upon logging into openstack-dashboard I failed to see my Nova project.

Lastly, go ahead and start up the server:

# tools/with_venv.sh dashboard/manage.py runserver 0.0.0.0:8000

At this point, you should be able to access your dashboard on http://x.x.x.x:8000, replacing x.x.x.x with your openstack-dashboard server's IP.

If you run into any issues, refer to this, which contains valid information for this particular version of the dashboard.

Again, running with this old version of the dashboard isn't ideal, and you certainly don't want to run it as root, but hopefully this will point you in the right direction if you struggle to get the latest version to work with Nova. My next task will to be to get a version of the dashboard and Nova which work together installed and operational.

First things first. Here’s the eth1 (private network) addresses assigned to my cloud servers:

nova-cc (our Nova cloud controller node):

eth1: 10.176.65.54

nova-compute (our Nova compute note, which will run our instances (QEMU or UML):

eth1: 10.176.95.220

Similar to previous posts, I went ahead and used 192.168.0.0/16 for Nova network as I didn’t have public IPs, nor did I want to interfere with the 10.176.64.0/18 network which is already used by this cloud provider.

On nova-cc, we need to install mysqld. Nova defaults to using SQLite, which works great when everything is running off a single instance. However, now that we’ve got another instance that needs to talk to nova-cc, we need a SQL server that it can connect to.

# apt-get update
# apt-get install mysql-server

Once installed, edit /etc/mysql/my.cnf and change this from:

bind-address            = 127.0.0.1

to:

bind-address            = 10.176.65.54

Finally, restart mysqld:

# service mysql restart

Now hop into the mysql shell and create a database and user/password to connect with:

# mysql -u root
mysql> CREATE DATABASE nova;
mysql> GRANT ALL PRIVILEGES ON nova.* TO nova@10.176.65.54 IDENTIFIED BY 'somepasshere';
mysql> GRANT ALL PRIVILEGES ON nova.* TO nova@10.176.95.220 IDENTIFIED BY 'somepasshere';

On nova-compute, the only nova-related package you really need is nova-compute:

# apt-get -y install python-software-properties
# add-apt-repository ppa:nova-core/milestone
# apt-get update
# apt-get install nova-compute

On both nova-cc and nova-compute:

# cat >> /etc/nova/nova.conf << "EOF"
--sql_connection=mysql://somepasshere@10.176.65.54/nova
--image_service=nova.image.glance.GlanceImageService
--glance_api_servers=10.176.65.54:9292
--rabbit_host=10.176.65.54
EOF
# for SERVICE in `ls -1 /etc/init.d/nova*`; do service $SERVICE restart; done

Now what we need to do is create our 192.68.0.0/16 network. We’ll use OpenVPN to do this, and we’ll use the eth1 private network (10.176.64.0/18) to do this this. Again, the idea here is to have a completely separate network which won’t interfere with what’s already out there.

On both:

# apt-get install openvpn

On nova-cc:

# cd /etc/openvpn
# openvpn --genkey --secret openvpn.key
# scp openvpn.key root@10.176.95.220:/etc/openvpn
# cat > /etc/openvpn/openvpn.server << "EOF"
dev tap
ifconfig 192.168.0.1 255.255.255.0
secret /etc/openvpn/openvpn.key
daemon
EOF
# cat > /etc/network/if-pre-up.d/00openvpn << "EOF"
#!/bin/bash
 
/usr/sbin/openvpn --config /etc/openvpn/openvpn.server
 
exit 0
EOF
# chmod 755 /etc/network/if-pre-up.d/00openvpn

On nova-compute:

# cat > /etc/openvpn/openvpn.client << "EOF"
remote 10.176.65.54
dev tap
ifconfig 192.168.0.3 255.255.255.0
secret /etc/openvpn/openvpn.key
daemon
EOF
# cat > /etc/network/if-pre-up.d/00openvpn << "EOF"
#!/bin/bash
 
/usr/sbin/openvpn --config /etc/openvpn/openvpn.client
 
/usr/sbin/brctl addbr br100
/usr/sbin/brctl addif br100 tap0
 
/sbin/ifconfig tap0 0.0.0.0
/sbin/ifconfig br100 192.168.0.3
 
exit 0
EOF
# chmod 755 /etc/network/if-pre-up.d/00openvpn

Finally, on both nodes:

# echo "--flat_interface=tap0" >> /etc/nova/nova/.conf

This causes nova-network to bridge into tap0.

Let’s recap what we’ve done. On nova-cc, we’re configuring OpenVPN to act as a server. We’re bringing tap0 up with IP 192.168.0.1/24 and the /etc/network/if-pre-up.d/00openvpn script ensures that the VPN server is started on on boot (specifically, before the other network devices are brought up). On nova-compute, we configure OpenVPN as a client, and the /etc/network/if-pre-up.d/00openvpn script creates a bridge (br100), adds the tap0 interface to it, and then brings 192.168.0.3 up on br100. If I recall correctly, the tap0 device doesn’t appear to be “up” until we ifconfig it, which is why we just set it to 0.0.0.0. Don’t quote me on this though, as I can’t quite remember.

I know very little about bridging, but essentially a bridge “connects two or more different physical ethernets together to form one large (logical) ethernet” (taken from /usr/share/doc/bridge-utils/HOWTO), and this is precisely what we have done here. We bridge the virtual interfaces for running instances (ie. vnet0) with tap0 (our VPN connection), which means that nova-cc can speak to instances running on nova-compute, and vice-versa. This is also essential as dnsmasq (our DHCP server) runs on nova-cc (spawned by nova-network), and without this bridging in place our instances would not be able to have their networking configured automatically on boot by the DHCP server.

Also, the reason why we don’t have to explicitly configure br100 on nova-cc is because that runs nova-network, which handles the bridging automatically. The only thing we did need to do on the nova-cc side is instruct nova-network on which device to bridge into (–flat_interface=tap0). The last thing I’ll say here is that OpenVPN used device tun0 by default, but we have to use tap0 (a virtual Ethernet adapter) as brctl creates Ethernet bridges, and a tun device is a “virtual point-to-point” link (see this for a tad more info).

Go ahead and reboot each instance, one at a time, to ensure that everything comes up as expected.

Once back up, on nova-cc:

# mysql -u root nova
mysql> SELECT * FROM fixed_ips WHERE id=4;
+---------------------+---------------------+------------+---------+----+-------------+------------+-------------+-----------+--------+----------+
| created_at          | updated_at          | deleted_at | deleted | id | address     | network_id | instance_id | allocated | leased | reserved |
+---------------------+---------------------+------------+---------+----+-------------+------------+-------------+-----------+--------+----------+
| 2011-07-22 16:21:35 | 2011-07-22 20:48:26 | NULL       |       0 |  4 | 192.168.0.3 |          1 |        NULL |         0 |      0 |        0 |
+---------------------+---------------------+------------+---------+----+-------------+------------+-------------+-----------+--------+----------+
1 row in set (0.00 sec)
 
mysql> UPDATE fixed_ips SET reserved=1 WHERE id=4 LIMIT 1;

What we’re doing here is “reserving” 192.168.0.3 for the other end of the VPN link on nova-compute. 192.168.0.2 is already reserved, but I’m not sure if nova uses this or will use it for something at some point. As such, just play it safe and reserve another available IP.

In theory, if you now launch an instance on nova-cc, it should build on nova-compute and the IP assigned should be accessible via nova-cc. The instance on nova-compute will have a gateway of 192.168.0.1 (which is physically on nova-cc), which means that all traffic in and out of the instance will travel through nova-cc. This also means that if nova-cc goes down, instances will not be able to communicate with the outside world (or potentially each other, though I’ve not tested myself).

That should be able it. I’ve probably missed a few things, but the general gist should be here. Also, I’m aware that there are no security best-practices implemented here, but the idea is to just get everything up and running as a proof of concept, and fine-tune later.

2011-07-13 21:15:22,256 INFO nova.virt.libvirt_conn [-] instance instance-0000003d: injecting key into image 3
2011-07-13 21:15:22,256 DEBUG nova.utils [-] Running cmd (subprocess): sudo losetup --find --show /var/lib/nova/instances/instance-0000003d/disk from (pid=838) execute /usr/lib/pymodules/python2.7/nova/utils.py:143
2011-07-13 21:15:22,424 DEBUG nova.utils [-] Running cmd (subprocess): sudo kpartx -a /dev/loop0 from (pid=838) execute /usr/lib/pymodules/python2.7/nova/utils.py:143
2011-07-13 21:15:22,509 DEBUG nova.utils [-] Running cmd (subprocess): sudo kpartx -d /dev/loop0 from (pid=838) execute /usr/lib/pymodules/python2.7/nova/utils.py:143
2011-07-13 21:15:22,563 DEBUG nova.utils [-] Running cmd (subprocess): sudo losetup --detach /dev/loop0 from (pid=838) execute /usr/lib/pymodules/python2.7/nova/utils.py:143
2011-07-13 21:15:22,604 WARNING nova.virt.libvirt_conn [-] instance instance-0000003d: ignoring error injecting data into image 3 (Mapped device was not found (we can only inject raw disk images): /dev/mapper/loop0p1)

I tried running the kpartx commands above, but they didn’t return anything. This was because the image I was using had no partition table.

To fix, I effectively created a new image and copied data from the original one.

To begin, create a spare file (see this for more info):

# cd /root
# dd if=/dev/zero of=CentOS5.6-AMD64-new-root_fs bs=1 count=0 seek=1024M

Now, create a partition to span the entire disk (replace /dev/loop0 with whatever losetup returns):

# losetup --show --find CentOS5.6-AMD64-new-root_fs
/dev/loop0
# fdisk /dev/loop0

Now, use kpartx to make the partition visible to the host, and create a filesystem on that partition:

# parted
# kpartx -a /dev/loop0
# mke2fs -j /dev/mapper/loop0p1

Mount the original image and copy data over:

# losetup --show --find CentOS5.6-AMD64-root_fs 
/dev/loop1
# mkdir /mnt/loop{0,1}
# mount /dev/mapper/loop0p1 /mnt/loop0
# mount /dev/loop1 /mnt/loop1
# cd /mnt/loop1
# rsync -a . /mnt/loop0

Update the fstab on the new image (this is necessary as the partition layout has now changed):

# cd /mnt/loop0/etc
# sed -i 's/ubda/ubda1/g' fstab
# cd /
# umount /mnt/loop{0,1}
# kpartx -d /dev/loop0
# losetup -d /dev/loop{0,1}

Modify /etc/nova/libvirt.xml.template, changing this line from:

<root>/dev/ubda</root>

to:

<root>/dev/ubda1</root>

That should be about it.

First things first. Let’s get install user-mode-linux:

# apt-get update
# apt-get install user-mode-linux

Now, grab the CentOS 5.6 x86_64 image from http://fs.devloop.org.uk/ (we’re using a 64-bit XenServer VM):

# cd /root
# wget http://fs.devloop.org.uk/filesystems/CentOS-5.6/CentOS5.6-AMD64-root_fs.bz2
# bunzip2 CentOS5.6-AMD64-root_fs.bz2

There is an image on http://wiki.openstack.org/Nova/UML, however I couldn’t get this image to boot properly. As such, I opted for the CentOS 5.6 image above.

A few things within the image needed adjusting, so I:

# mkdir /mnt/image
# losetup --find --show CentOS5.6-AMD64-root_fs
/dev/loop0
# mount /dev/loop0 /mnt/image
# cp -a /usr/lib/uml/modules/2.6.35.1/ /mnt/image/lib/modules/
# chroot /mnt/image
# sed -i 's@LABEL=ROOT@/dev/ubda@g' /etc/fstab
# chkconfig network on
# exit
# umount /mnt/image
# losetup -d /dev/loop0

Note that /usr/lib/uml/modules/2.6.35.1/ is provided by the user-mode-linux package on Ubuntu 10.10, so adjust accordingly to what you’re running. Also, UML seems to use device /dev/ubda, so we modify /etc/fstab w/ that.

Now we can bundle up the image:

# cd /root
# euca-bundle-image -i CentOS5.6-AMD64-root_fs
# euca-upload-bundle -b uml-image-bucket -m /tmp/CentOS5.6-AMD64-root_fs.manifest.xml
# euca-register uml-image-bucket/CentOS5.6-AMD64-root_fs.manifest.xml

Once that’s done, we remove this from /etc/nova/nova.conf:

--libvirt_type=qemu

… and add this:

--libvirt_type=uml
--use_cow_images=false
--libvirt_xml_template=/etc/nova/libvirt.xml.template

Since I’ve specified –libvirt_xml_template in /etc/nova/nova.conf, we need to now create that file:

# cp -a /usr/share/pyshared/nova/virt/libvirt.xml.template /etc/nova/libvirt.xml.template

Now open up /etc/nova/libvirt.xml.template and remove the following:

#if $getVar('vncserver_host', False)
        <graphics type='vnc' port='-1' autoport='yes' keymap='en-us' listen='${vncserver_host}'/>
#end if

The reason for doing this is because it appears that we run into an issue similar to the one reported in here. Update: I’ve since created a bug report for this UML issue here.

Now we can restart the nova-compute service and create an instance:

# service nova-compute restart
# euca-run-instances ami-778c501e -k mykey -t m1.tiny

(replace ami-778c501e with your image name, which can be found by running euca-describe-images)

If your instance doesn’t go into a running state, have a look at the libvirt.xml file under /var/lib/nova/instances/####/ (replacing #### w/ your instance’s name, found by running euca-describe-instances), ensuring that there is no reference to the vnc stuff in there. If there is, then the template hasn’t been updated or isn’t being used correctly. Otherwise, your instance should be ssh-able, using the IP returned by euca-describe-instances.

http://wiki.openstack.org/NovaInstall/DevPkgInstall
http://wiki.openstack.org/RunningNova

However, I needed some additional steps to get this working in my virtualized environment.

Anyway, let’s get started. First off, get the VM’s package index and installed packages updated:

# apt-get update
# apt-get upgrade

Now we start with the Nova installation:

# apt-get install rabbitmq-server
# apt-get install python-software-properties
# add-apt-repository ppa:nova-core/milestone
# apt-get update
# apt-get install python-nova
# apt-get install nova-common nova-doc nova-api nova-network nova-objectstore nova-scheduler nova-compute euca2ools unzip

A listing of available OpenStack PPAs is available here. I’ve opted to use ppa:nova-core/milestone, which is the “last development milestone”.

Instead of creating a file for volumes (as outlined in http://wiki.openstack.org/RunningNova), I create a partition at the end of my disk (since there’s free space), and use that instead. Note that I set the partition to Linux LVM (8e), since we’ll be using LVM.

# fdisk /dev/sda
# partprobe
# apt-get install lvm2
# vgcreate nova-volumes /dev/sda2

Update: This volume doesn’t appear to get used when creating images of type qemu or uml.

Now we create a network to be used by our instances. I didn’t use 10.0.0.0/8 as per http://wiki.openstack.org/RunningNova, as my VM is already connected to a subnet in that network, so I use 192.168.0.0/16 instead.

# nova-manage network create 192.168.0.0/16 1 256

This creates:

# nova-manage network list
network           	netmask        	start address  	DNS            
192.168.0.0/24    	255.255.255.0  	192.168.0.3    	None           
#

What we now do is create an admin user, create a project, and unzip nova.zip into /root. Sourcing novarc sets up all our environment variables, and will need to be run each time we log out and back into the host.

# cd /root
# nova-manage user admin mattt
# nova-manage project create test mattt
# nova-manage project zipfile test mattt
# unzip nova.zip 
Archive:  nova.zip
 extracting: novarc                  
 extracting: pk.pem                  
 extracting: cert.pem                
 extracting: cacert.pem              
# . novarc

Now we grab a Linux image to use for our instances, and register it:

# cd /root
# wget http://uec-images.ubuntu.com/releases/10.04/release/ubuntu-10.04-server-uec-amd64.tar.gz
# uec-publish-tarball ubuntu-10.04-server-uec-amd64.tar.gz mybucket
Fri Jul  1 09:23:53 UTC 2011: ====== extracting image ======
Warning: no ramdisk found, assuming '--ramdisk none'
kernel : lucid-server-uec-amd64-vmlinuz-virtual
ramdisk: none
image  : lucid-server-uec-amd64.img
Fri Jul  1 09:24:12 UTC 2011: ====== bundle/upload kernel ======
Fri Jul  1 09:24:14 UTC 2011: ====== bundle/upload image ======
Fri Jul  1 09:25:23 UTC 2011: ====== done ======
emi="ami-3d4b2b15"; eri="none"; eki="aki-4eb969b8";
#

We should now be able to list the image:

# euca-describe-images 
IMAGE	aki-4eb969b8	mybucket/lucid-server-uec-amd64-vmlinuz-virtual.manifest.xml		available	public		x86_64	kernel		
IMAGE	ami-3d4b2b15	mybucket/lucid-server-uec-amd64.img.manifest.xml		available	public		x86_64	machine	aki-4eb969b8	
#

Note that when using ppa:nova-core/trunk, I had to do the following to get this to work:

# apt-get install glance
# glance-control api start

I’ll need to circle back to ppa:nova-core/trunk later and see why the default install relies on glance, but doesn’t install it.

Now we create an ssh key to inject into our images, and we go ahead and spawn an instance. We use the ami for the image registered above (ami-3d4b2b15 in this case):

# euca-add-keypair mykey > mykey.priv
# chmod 600 mykey.priv
# euca-run-instances ami-3d4b2b15 -k mykey -t m1.tiny
RESERVATION	r-hcvy99xp	test	default
INSTANCE	i-00000002	ami-3d4b2b15			scheduling	mykey (test, None)	0		m1.tiny	2011-07-01T09:33:39Z	unknown zone
#

We should now be able to list our instances, and see the status as “running”:

# euca-describe-instances
RESERVATION	r-hcvy99xp	test	default
INSTANCE	i-00000002	ami-3d4b2b15	192.168.0.4	192.168.0.4	running	mykey (test, nova-blog-1gb)	0		m1.tiny	2011-07-01T09:33:39Z	nova
#

Note that my original instance creation failed, as it was trying to (by default) create a KVM instance. Since my hardware doesn’t support KVM, and I’m running within a VM already, I had to append this to /etc/nova/nova.conf:

# echo "--libvirt_type=qemu" >> /etc/nova/nova.conf
# service nova-compute restart

We should in theory be able to use UML instead of qemu (see http://wiki.openstack.org/Nova/UML), but I haven’t fully tested this. More information to come.

My instance should now be accessible over the network:

# ping -c 1 192.168.0.4
PING 192.168.0.4 (192.168.0.4) 56(84) bytes of data.
64 bytes from 192.168.0.4: icmp_req=1 ttl=64 time=0.000 ms
 
--- 192.168.0.4 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.000/0.000/0.000/0.000 ms
#

The IP for the instance can be found via euca-describe-instances executed above.

Assuming the instance is pingable, I should be able to ssh into it:

# euca-authorize -P tcp -p 22 default
# ssh -i mykey.priv root@192.168.0.4

Note that I didn’t actually have to run euca-authorize in order to ssh into my instance, so I suspect the default security group allows all.

Lastly, you can terminate the instance:

# euca-terminate-instances i-00000002

The one final thing I will say is that in order for my instances to be able to access the public Internet, I had to add the following to /etc/nova/nova.conf (replacing x.x.x.x w/ my VM’s public IP address):

# echo "--routing_source_ip=x.x.x.x" >> /etc/nova/nova.conf
# echo "--fixed_range=192.168.0.0/24" >> /etc/nova/nova.conf
# service nova-network restart

This causes the correct POSTROUTING chain SNAT rules in the nat table to get created, allowing my instance on the 192.168.0.0/24 network to reach the public Internet.

Please let me know if you see any glaringly obvious errors here, and again 99% of this information was obtained from http://wiki.openstack.org/NovaInstall/DevPkgInstall and http://wiki.openstack.org/RunningNova. I haven’t given thorough info on what all the commands do as quite frankly I don’t know myself, but will circle back once I have more knowledge on everything.

Using my Rackspace Cloud Servers account, I fired up a Fedora 14 VM and once logged in ran:

# mkdir -p /xen/var/lib/rpm
# mkdir -p /xen/var/log
# touch /xen/var/log/yum.log
# rpm --root /xen --initdb

I then installed the Fedora 14-specific release package:

# rpm -ihv --nodeps --root /xen fedora-release-14-1.noarch

Finally, I went ahead and installed all Core packages (a more minimal package set than Base, which is what was used in the original post):

# yum --installroot=/xen -y groupinstall Core

At this point, I tar’d up /xen and copied over to my xen dom0. Sadly, this Fedora 14 domU wouldn’t boot w/ the vmlinuz-2.6.26-2-xen-amd64 kernel on Debian Lenny, but fortunately I had access to a newer pv-ops kernel that does work. Once I copied kernel modules into the domU and did some final last configurations (setting root password, modifying /etc/fstab), the VM was bootable and appeared to work fine.

In retrospect, I could probably remove some unnecessary packages such as kernel, grub, and so on, but that’s pretty insignificant and aren’t worth messing about with at this point in time.