Long time, no write. This will change immediately.
Recently, I have getting myself familiar with OpenStack by way of the book OpenStack Cloud Computing Cookbook. For the most part, it was an easy read and I got my home OpenStack environment and running, albeit with some issues.
The book starts up with asking the reader to setup up a couple of Ubuntu 12.04 using VirtualBox. I decided to be clever and setup KVM instances instead, with fairly mixed results (At some point during the exploration, I decided to started over and went with some spare physical hardware - just as well, I needed to redo my private lab anyway).
Next, I went and installed the following prerequisites per instruction on one single server:
sudo apt-get -y install rabbitmq-server nova-api nova-objectstore nova-scheduler nova-network nova-compute nova-cert glance qemu unzip.
Then I setup pressed parameters for MySQL server. Originally, it was MySQL 5.1
cat MYSQL_PRESEED | debconf-set-selections<
mysql-server-5.1 mysql-server/root_password password openstack
mysql-server-5.1 mysql-server/root_password_again password openstack
mysql-server-5.1 mysql-server/start_on_boot boolean true
MYSQL_PRESEED
However, Ubuntu 12.04.02 (which is what I am using) apparently install 5.5 by default, so some quick changes was in order:
cat MYSQL_PRESEED | debconf-set-selections
mysql-server-5.5 mysql-server/root_password password openstack
mysql-server-5.5 mysql-server/root_password_again password openstack
mysql-server-5.5 mysql-server/start_on_boot boolean true
MYSQL_PRESEED
sudo apt-get update
sudo apt-get -y install mysql-server
sudo sed -i 's/127.0.0.1/0.0.0.0/g' /etc/mysql/my.cnf
MYSQL_PASS=openstack
mysql -uroot -p$MYSQL_PASS -e 'CREATE DATABASE nova;'
mysql -uroot -p$MYSQL_PASS -e "GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'%'"
mysql -uroot -p$MYSQL_PASS -e "SET PASSWORD FOR 'nova'@'%' = PASSWORD('$MYSQL_PASS');"
--sql_connection=mysql://nova:openstack@192.168.15.105/nova
The IP .105 being the IP of the Ubuntu box openstack1.
Then I added the next set of parameters:
--use_deprecated_auth
--s3_host=192.168.15.105
--rabbit_host=192.168.15.105
--ec2_host=192.168.15.105
--ec2_dmz_host=192.168.15.105
--public_interface=eth1
--image_service=nova.image.glance.GlanceImageService
--glance_api_servers=192.168.15.105:9292
--auto_assign_floating_ip=true
--scheduler_default_filters=AllHostsFilter
sudo nova-manage db sync
Then i setup the network ranges.
sudo nova-manage network create vmnet --fixed_range_v4=10.0.0.0/8 --network_size=64 --bridge_interface=eth0
sudo nova-manage floating create --ip_range=192.168.15.0/24
The first set is for administrative access VMs to talk to each other as well to OpenStack. The other set is the public facing IP range (or at least public from a user standpoint).
Then I stopped and started the following services:
nova-compute
nova-network
nova-api
nova-scheduler
nova-objectstore
nova-cert
libvirt-bin
glance-registry
glance-api
By way of brief explanation, nova-compute creates and destroy the instances, while nova-manage assigns and creates the IPs and VLANs. The nova-api provides access to services by the various application components, nova-scheduler runs the commands submitted by the various services.
I am not sure about nova-objectstore, but I am sure that nova-vert encrypts the connection between the various services. libvirt-bin, of course, is the wrapper around KVM and provide access to the KVM hypervisor. Finally, glance-registry and glance-api registers and manage the images.
At this point, I went and created a user, gave it admin access, and then created a project called “cookbook”:
sudo nova-manage user admin openstack
sudo nova-manage role add openstack cloudadmin
sudo nova-manage project create cookbook openstack
Then I zip up the cook:
sudo nova-manage project zipfile cookbook openstack
Then installed the tools on my Ubuntu) client necessary to manage openstack
sudo apt-get install euca2ools python-novaclient unzip
(later one, I used the Mac OS X version, which I will get to later on).
Copied the zip file from openstack1, unzip it into a directory called openstack, cd to it, then source an environment file so that I got the parameters into the shell.
. novarc
Finally, I generated a key and inserted into the database
nova keypair-add openstack > openstack.pem
chmod 0600 *.pem
This allows me to setup a SSH key to whatever the default user of the instance I am created, so that I can simply run:
ssh -i openstack.pem username@instance name.
(the book had it was chmod 0600.pem, BTW, which is an obvious typo)
Finally, I was ready to update my image. So I downloaded a cloud version of Ubuntu’s server:
wget http://uec-images.ubuntu.com/releases/precise/release/ubuntu-12.04-server-cloudimg-i386.tar.gz
Installed the cloud_util tools:
sudo apt-get install cloud_utils
And then I attempted to upload the image …
cloud-publish-tarball ubuntu-12.04-server-cloudimg-i386.tar.gz images i386
and I ran into problems - kept running of space. I though the destination was the issue, so I re-did the server (at the time, it was a virtual server, not a physical). Eventually, turns out that it is running out space at the source - I had installed the Ubuntu server partitioned with separate file systems for /, /usr, /var, /tmp. By default, it extracts to /tmp, which was too small. So I change the default in the shell to:
TMPDIR=/var/tmp
TEMPDIR=/var/tmp
export TMPDIR ; export TEMPDIR
And afterwards, I was able to upload the image, which I was able to view with:
++-+++
| ID | Name | Status | Server |
++-+++
| 6e63d0cd-4e24-4766-ad48-3a01670a607e | images/precise-server-cloudimg-i386-vmlinuz-virtual | ACTIVE | |
| bedf0e78-c7d4-414e-85fb-291a0ccd851d | images/precise-server-cloudimg-i386.img | ACTIVE | |
++-+++
So I ready to build an instance.
I added the appropriate access to the ports:
nova secgroup-add-rule default tcp 22 22 0.0.0.0/0
nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0
And then I went and boot or create my instances:
nova boot myInstance image 0e2f43a8-e614-48ff-92bd-be0c68da19f4 flavor 2 key_name openstack
And… it didn’t quite work. I mean, I can see the console output with:
nova console-log myInstance
But no IP was assigned, so I couldn’t login. After much reason, I found that the firewall rules was proving DHCP request from going through, so I added:
iptables -A POSTROUTING -t mangle -p udp dport 68 -j CHECKSUM checksum-fill
And from that point on, DHCP requests was going through and I was able to login.
At this point, I felt confident and decided to skip ahead and added another node to the group.
Big mistake. But I am getting ahead of myself.
I went to chapter 11 and per instruction, installed just the following:
sudo apt-get -y installed nova-compute nova-network nova-apit
Copied the nova.conf to the new node, updating the IPs, verified on the original node that the services are listening:
root@openstack1:~# nova-manage service list
2013-04-27 22:38:46 DEBUG nova.utils [req-9dd11667-7967-4221-807f-98ddaf9371b3 None None] backend from (pid=18080) __get_backend /usr/lib/python2.7/dist-packages/nova/utils.py:662
Binary Host Zone Status State Updated_At
nova-cert openstack1 nova enabled :-) 2013-04-28 03:38:45
nova-scheduler openstack1 nova enabled :-) 2013-04-28 03:38:44
nova-network openstack1 nova enabled :-) 2013-04-28 03:38:45
nova-compute openstack1 nova enabled :-) 2013-04-28 03:38:38
nova-compute openstack2 nova enabled :-) 2013-04-28 03:38:39
nova-network openstack2 nova enabled :-) 2013-04-26 16:58:30
Feeling pretty sure that it will work as intended, I attempted to create more instances - and I failed. IPs failed to be assigned once again.
First of all, I didn’t setup my switch properly to handle the private network between the new nodes. I forgot my switch can only separate ports into separate broadcast domains using VLAN tagging (which OpenStack uses by default by way of VLAN Manager, which I didn’t pay much attention to - this will become significant shortly in this blog). After a while, I just gave up and plugged a crossover between openstack1 and openstack2 (later one, I did remember how to setup the switch properly and got the packets tagged appropriately).
So at that point, I was able to create the instances. But then I couldn’t login using my private keys. Reviewing my console log, I found this (example pulled from google):
http://169.254.169.254/2009-04-04/meta-data/instance-id failed [50/120s]: url error [timed out]
When you build an instance, it pulls the credentials from this loop back, which in turns is supposed to be routed to the API on the controller (which is on openstack1). I corrected that problem by adding the route:
sudo ip route add 169.254.0.0/16 metric 1000 dev eth1
openstack@openstack2:~$ ip route show
default via 192.168.15.1 dev eth1 metric 100
169.254.0.0/16 dev eth1 scope link metric 1000
192.168.15.0/24 dev eth1 proto kernel scope link src 192.168.15.110
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
But the problem still persists.
Finally, after several long looks at the following pages:
http://www.mirantis.com/blog/openstack-networking-single-host-flatdhcpmanager/
http://www.mirantis.com/blog/openstack-networking-flatmanager-and-flatdhcpmanager/
And going back and reading at chapter 10 in the book (the chapter I skipped) I uninstalled nova-network. And suddenly, the instances are able to reach the API now.
Remember how the default OpenStack setup was using VLAN Manager? That is useful if you need to separate tenants using IP ranges and VLANs. More importantly, it meant that nova-network is only needed on the controller side (since the controller handles the route and IP. It is only when I need to use the other networking setup (Flat Networking or Flat Networking with DHC - where I isolate the tenants using Security Group Modes) is nova-network is necessary on the new nodes. Otherwise, the firewalls setup by nova-network was blocking access to the API by the instances.
So that was resolved and I was able to build instances at will - well mostly. Just tried to create a new instance just now. and I get this:
ERROR: Quota exceeded: code=InstanceLimitExceeded (HTTP 413) (Request-ID: req-82aa560f-0318-4e55-b5bd-98b10d1b9c60)
Heh. Removing one instance now:
stardust:openstack rilindo$ nova delete vmi001
stardust:openstack rilindo$ nova list
+++++
| ID | Name | Status | Networks |
+++++
| 7a6a4912-71ce-45e2-8a38-51537a4c7ffb | vmi001 | ACTIVE | vmnet=10.0.0.4, 192.168.15.12 |
| 2fbaad51-35f4-4ac4-b6a3-d338af5905d8 | vmi002 | ACTIVE | vmnet=10.0.0.7, 192.168.15.13 |
| c70c7522-c8fb-4196-8161-6c6153549729 | vmi003 | ACTIVE | vmnet=10.0.0.8, 192.168.15.16 |
| 9899b0ee-4cac-4588-891b-705a6cc95512 | vmi004 | ACTIVE | vmnet=10.0.0.9, 192.168.15.17 |
| 41db335c-97a2-4335-9e91-7a0a4345b70a | vmi005 | ACTIVE | vmnet=10.0.0.10, 192.168.15.18 |
| 4d2698e2-88ba-42f6-9107-0380d89c3e89 | vmi006 | ACTIVE | vmnet=10.0.0.11, 192.168.15.19 |
| 9cf60a41-1a58-4dbd-a47c-b4f2c42e6117 | vmi007 | ACTIVE | vmnet=10.0.0.12, 192.168.15.24 |
| de257ab1-cf03-4f08-816f-c9e16ce2793a | vmi008 | ACTIVE | vmnet=10.0.0.13, 192.168.15.25 |
| 0c67d3bc-1ca1-4cfe-9653-eb3a5c94350f | vmi009 | ACTIVE | vmnet=10.0.0.14, 192.168.15.26 |
| 9b5601bf-969f-447f-9e2a-8727dd3d45e2 | vmi010 | ACTIVE | vmnet=10.0.0.15, 192.168.15.27 |
+++++
stardust:openstack rilindo$ nova list
+++++
| ID | Name | Status | Networks |
+++++
| 2fbaad51-35f4-4ac4-b6a3-d338af5905d8 | vmi002 | ACTIVE | vmnet=10.0.0.7, 192.168.15.13 |
| c70c7522-c8fb-4196-8161-6c6153549729 | vmi003 | ACTIVE | vmnet=10.0.0.8, 192.168.15.16 |
| 9899b0ee-4cac-4588-891b-705a6cc95512 | vmi004 | ACTIVE | vmnet=10.0.0.9, 192.168.15.17 |
| 41db335c-97a2-4335-9e91-7a0a4345b70a | vmi005 | ACTIVE | vmnet=10.0.0.10, 192.168.15.18 |
| 4d2698e2-88ba-42f6-9107-0380d89c3e89 | vmi006 | ACTIVE | vmnet=10.0.0.11, 192.168.15.19 |
| 9cf60a41-1a58-4dbd-a47c-b4f2c42e6117 | vmi007 | ACTIVE | vmnet=10.0.0.12, 192.168.15.24 |
| de257ab1-cf03-4f08-816f-c9e16ce2793a | vmi008 | ACTIVE | vmnet=10.0.0.13, 192.168.15.25 |
| 0c67d3bc-1ca1-4cfe-9653-eb3a5c94350f | vmi009 | ACTIVE | vmnet=10.0.0.14, 192.168.15.26 |
| 9b5601bf-969f-447f-9e2a-8727dd3d45e2 | vmi010 | ACTIVE | vmnet=10.0.0.15, 192.168.15.27 |
+++++
And adding a new one:
stardust:openstack rilindo$ nova boot vmi011 image bedf0e78-c7d4-414e-85fb-291a0ccd851d flavor 2 key_name mykey
+-++
| Property | Value |
+-++
| status | BUILD |
| updated | 2013-04-28T04:02:23Z |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-SRV-ATTR:host | openstack2 |
| key_name | mykey |
| image | images/precise-server-cloudimg-i386.img |
| hostId | e1693fb6dfb89d758273a4312096678745f8f568dbdc3fbe279e286b |
| OS-EXT-STS:vm_state | building |
| OS-EXT-SRV-ATTR:instance_name | instance-00000046 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| flavor | m1.small |
| id | 52cc6451-1c5c-462e-8d06-6f9465ce1a94 |
| user_id | openstack |
| name | vmi011 |
| adminPass | RVeaJd7j7Kmu |
| tenant_id | cookbook |
| created | 2013-04-28T04:02:22Z |
| OS-DCF:diskConfig | MANUAL |
| accessIPv4 | |
| accessIPv6 | |
| progress | 0 |
| OS-EXT-STS:power_state | 0 |
| metadata | {} |
| config_drive | |
+-++
stardust:openstack rilindo$ nova list
+++++
| ID | Name | Status | Networks |
+++++
| 2fbaad51-35f4-4ac4-b6a3-d338af5905d8 | vmi002 | ACTIVE | vmnet=10.0.0.7, 192.168.15.13 |
| c70c7522-c8fb-4196-8161-6c6153549729 | vmi003 | ACTIVE | vmnet=10.0.0.8, 192.168.15.16 |
| 9899b0ee-4cac-4588-891b-705a6cc95512 | vmi004 | ACTIVE | vmnet=10.0.0.9, 192.168.15.17 |
| 41db335c-97a2-4335-9e91-7a0a4345b70a | vmi005 | ACTIVE | vmnet=10.0.0.10, 192.168.15.18 |
| 4d2698e2-88ba-42f6-9107-0380d89c3e89 | vmi006 | ACTIVE | vmnet=10.0.0.11, 192.168.15.19 |
| 9cf60a41-1a58-4dbd-a47c-b4f2c42e6117 | vmi007 | ACTIVE | vmnet=10.0.0.12, 192.168.15.24 |
| de257ab1-cf03-4f08-816f-c9e16ce2793a | vmi008 | ACTIVE | vmnet=10.0.0.13, 192.168.15.25 |
| 0c67d3bc-1ca1-4cfe-9653-eb3a5c94350f | vmi009 | ACTIVE | vmnet=10.0.0.14, 192.168.15.26 |
| 9b5601bf-969f-447f-9e2a-8727dd3d45e2 | vmi010 | ACTIVE | vmnet=10.0.0.15, 192.168.15.27 |
| 52cc6451-1c5c-462e-8d06-6f9465ce1a94 | vmi011 | ACTIVE | vmnet=10.0.0.4, 192.168.15.12 |
+++++
stardust:openstack rilindo$ ssh -i mykey.pem ubuntu@192.168.15.12
The authenticity of host 192.168.15.12 (192.168.15.12) cant be established.
RSA key fingerprint is 85:01:59:17:8d:11:4b:7c:60:72:c8:09:be:2d:45:73.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 192.168.15.12 (RSA) to the list of known hosts.
Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.2.0-40-virtual i686)
* Documentation: https://help.ubuntu.com/
System information as of Sun Apr 28 04:17:07 UTC 2013
System load: 0.0 Processes: 61
Usage of /: 6.9% of 9.84GB Users logged in: 0
Memory usage: 1% IP address for eth0: 10.0.0.4
Swap usage: 0%
Graph this data and manage this system at https://landscape.canonical.com/
Get cloud support with Ubuntu Advantage Cloud Guest:
http://www.ubuntu.com/business/services/cloud
Use Juju to deploy your cloud instances and workloads:
https://juju.ubuntu.com/#cloud-precise
0 packages can be updated.
0 updates are security updates.
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
To run a command as administrator (user root), use sudo .
See man sudo_root for details.
ubuntu@vmi011:~$
URLs I used as reference:
http://www.mail-archive.com/openstack@lists.launchpad.net/msg20584.html
http://www.mirantis.com/blog/openstack-networking-single-host-flatdhcpmanager/
http://www.mirantis.com/blog/openstack-networking-flatmanager-and-flatdhcpmanager/
http://www.gossamer-threads.com/lists/openstack/dev/17681
http://howtobyexamples.blogspot.com/2011/10/openstack-novaconf-configuration.html
http://blog.tocisoft.com/2011/06/why-ec2-ami-tools-ec2-upload-bundle.html
https://answers.launchpad.net/nova/+question/215096
http://docs.openstack.org/essex/openstack-compute/admin/content/network-troubleshooting.html
http://docs.openstack.org/folsom/openstack-network/admin/content/adv_cfg_l3_agent_metadata.html
Note: The book was written for Openstack Essex. Since then, Flosom and now Grizzly was released. Which mean that much of the results I found was for the latter, making things more for me, as I wasn’t sure if the solution was applicable to Essex or not. (For example, nova-network was replaced by Quantum, which apparently does networking a bit diffently).