Category Archives: Lab Stough

OpenStack Lab Installation with DevStack

I have been experimenting with OpenStack in a personal lab at home. My motivation has been to get some hands on exposure with Quantum. OpenStack is definitely not a trivial task to set up. I have built it with DevStack, a script that builds complete OpenStack development environments, and have encountered a few bumps along the road. The first one was of package dependencies. I have learned that unlike on our personal laptops running MacOS or Windows, when running a Linux server, such as Ubuntu, patching the kernel to the latest release is not always a good idea. In the case of DevStack, I ran into package dependency errors, which are a nightmare to resolve. There is no general consensus on forums for how to mitigate them.

The only way I was able to get around this was to run the DevStack script with an unaltered kernel from Ubuntu 12.04 LTS. That means no running of ‘apt-get update’ followed by ‘apt-get upgrade’. I was able to get OpenStack successfully running with Kernel 3.2.0-29, which is what Ubuntu 12.04 LTS comes with natively. I’ve attended a few OpenStack meetups and my experience is consistent with those of other attendees who I have interacted with. In hindsight, it is not surprising that DevStack broke because OpenStack has so many constant code changes and moving parts. DevStack, which pulls the latest release from Git Hub, is likely to break if a major variable change, such as kernel upgrade, is introduced.

I got some help from the inimitable Brent Salisbury, whose blog posts have come in handy on several occasions. Next I plan to add customizations to my OpenStack installation, such as adding Quantum plug-ins rather than using nova-network.

Handling Kernel Updates with Open vSwitch

Last month I wrote how I built Open vSwitch 1.4.0 package on Ubuntu 12.04. Immediately afterwards I left my lab and when I returned to it, nearly a month later, I ran an apt-get upgrade out of habit. Consequently, the kernel got upgraded from 3.2.0-34 to 3.2.0-36 and I ran into the following error when starting the OVS service:

root@pakdude-02:~# uname -a
Linux pakdude-02 3.2.0-36-generic #57-Ubuntu SMP Tue Jan 8 21:44:52 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
root@pakdude-02:~# service openvswitch-switch start
 FATAL: Module openvswitch_mod not found.
 * Inserting openvswitch module
 Module has probably not been built for this kernel.
 For instructions, read
 FATAL: Module openvswitch_mod not found.
 * Inserting openvswitch module

It turns out that this was because the kernel modules also needed to be updated. The following command did the trick:

root@pakdude-02:~# module-assistant auto-install openvswitch-datapath

I could have simply typed the following instead with the same consequences.

root@pakdude-02:~# m-a a-i openvswitch-datapath

As per the documentation, module-assistant aims to facilitate the process of building kernel modules from source. In other words, this needs to be run each time the kernel is upgraded.

Open vSwitch 1.4 installation from package on Ubuntu 12.04

In trying to get a more grounded feeling for OpenStack I’ve decided to build a home lab. One step involves configuring Open vSwitch to bridge with VMs. In this post I shall cover the Open vSwitch (OVS) build process along with KVM installation. Future posts shall cover more detailed configurations and scenarios along with videos.

While I am more familiar with the CentOS/RHE flavors of Linux, there seems to be more support for OVS on the Debian/Ubuntu platform. So in this post I am covering Ubuntu 12.04 LTS. There are two ways to install OVS:

  • Use Ubuntu’s apt-get installer to install packages – easier
  • Build from source code – more difficult

This post is aiming at the low-hanging fruit of building from the package. The drawback is that newer features are unavailable in the package. The package version of OVS is 1.4.0. The most stable Long Term release, as of writing, is 1.4.3, while the latest release, 1.7.1, includes support for VXLAN and Open Flow. I plan to document my findings with various builds and Linux flavors in future posts.

As I mentioned, I built OVS 1.4.0 off of Ubuntu 12.04 LTS (Long Term Support), which runs kernel version 3.2. The following steps are taken from various documents on the OVS site, while the outputs are excerpts from my lab.

root@pakdude-02:~# uname -a
Linux pakdude-02 3.2.0-34-generic #53-Ubuntu SMP Thu Nov 15 10:48:16 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
root@pakdude-02:~# apt-get install build-essential fakeroot openvswitch-switch openvswitch-common openvswitch-datapath-source

Keep in mind that additional packages, such as dkms (Dynamic Kernel Module Support), were installed as a result because they were pre-requisites.
The following output is good:

DKMS: build completed.

Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.2.0-34-generic/updates/dkms/

Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.2.0-34-generic/updates/dkms/


DKMS: install completed.
Setting up openvswitch-switch (1.4.0-1ubuntu1.3) ...
 * Inserting openvswitch module
 * /etc/openvswitch/conf.db does not exist
 * Creating empty database /etc/openvswitch/conf.db
 * Starting ovsdb-server
 * Configuring Open vSwitch system IDs
 * Starting ovs-vswitchd
 * Enabling gre with iptables

OVS has now been built. We will verify shortly. But first, we need to install KVM, a full-blown virtualization solution for Linux, and libvirt-bin, a daemon that loads the KVM modules. KVM also inclue virsh, which is a tool to manage (create, start, stop, etc) virtual domains or networks. Remember, KVM requires libvirt-bin.

root@pakdude-02:~# apt-get install libvirt-bin

Note that this will install bridge-utils and ebtables as well. We will get to that shortly. First, we want to destroy the default network created by libvirt-bin, which is virbr0. OVS will supply the network instead.

root@pakdude-02:~# ifconfig virbr0
virbr0    Link encap:Ethernet  HWaddr 4e:c0:0d:41:e3:0c  
          inet addr:  Bcast:  Mask:
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

root@pakdude-02:~# virsh net-destroy default
Network default destroyed

root@pakdude-02:~# virsh net-autostart --disable default
Network default unmarked as autostarted

root@pakdude-02:~# ifconfig virbr0
virbr0: error fetching interface information: Device not found

Now we have to actually install KVM.

root@pakdude-02:~# apt-get install kvm

Some additional packages are installed in the process.
Keep in mind that ebtables is not needed, so remove it. OVS will play the role of the bridge.

root@pakdude-02:~# apt-get purge ebtables

bridge still showed up in lsmod | grep bridge, but there was no need to rmmod it (as shown in many other guides on the web) as it was gone upon the next reboot. Remember, OVS will assume the bridging functionality. Some guides mention Bridge Compatibility installation. However, I do not see the need. Bridge Compatibility provides a way for applications that use the Linux bridge to gradually migrate to OVS. Programs that ordinarily control the Linux bridge module, such as brctl, instead control the OVS kernel-based switch. If you do not already depend on these programs, then you do not need bridge compatibility.

root@pakdude-02:~# service openvswitch-switch status
ovsdb-server is running with pid 1104
ovs-vswitchd is running with pid 1125
root@pakdude-02:~# ovs-vsctl show
    ovs_version: "1.4.0+build0"

We’re good to go. Additionally, these are the relevent processes that are now running:

root@pakdude-02:~# ps -face | grep ovs
root      1103     1 TS   29 23:45 ?        00:00:00 ovsdb-server: monitoring pid 1104 (healthy)                                                                                                                                                                                                                                                                                                                                                                       
root      1104  1103 TS   29 23:45 ?        00:00:00 ovsdb-server /etc/openvswitch/conf.db -vANY:CONSOLE:EMER -vANY:SYSLOG:ERR -vANY:FILE:INFO --remote=punix:/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,manager_options --private-key=db:SSL,private_key --certificate=db:SSL,certificate --bootstrap-ca-cert=db:SSL,ca_cert --no-chdir --log-file=/var/log/openvswitch/ovsdb-server.log --pidfile=/var/run/openvswitch/ --detach --monitor
root      1124     1 TS   29 23:45 ?        00:00:00 ovs-vswitchd: monitoring pid 1125 (healthy)                                                                                                                                                                                                 
root      1125  1124 TS   29 23:45 ?        00:00:00 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vANY:CONSOLE:EMER -vANY:SYSLOG:ERR -vANY:FILE:INFO --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ --detach --monitor
root      2346  2183 TS   19 23:57 pts/1    00:00:00 grep --color=auto ovs

And that’s about it. Hopefully I’ll get some functionality and configurations up here soon.