Category Archives: OpenStack

OpenStack Cloud Adoption Picking Up

I was listening to the recent Packet Pushers podcast Show 138. HP, who sponsored the show, spoke about HP Cloud, a public cloud that is built on OpenStack. It was launched in May 2012 and made generally available in December. HP is taking Amazon head on in the cloud provider space (at least when it comes to SLAs) as the GM of HP Cloud Services puts it. HP Cloud comprises multiple availability zones, identity management, account management, block storage (ala AWS S2). Compare that with the Cisco Edition of OpenStack, which is just a validated deployment, tested on Cisco’s UCS servers and Nexus switches only. In other words, Cisco does not offer an Enterprise-grade cloud like HP or Amazon do.

And now you can throw IBM’s name into that hat. This week IBM unveiled a new cloud offering based on open cloud standards, including OpenStack, that significantly speeds and simplifies managing an enterprise-grade cloud. IBM, like HP, is one of the founding members of OpenStack and is launching this OpenStack-based public cloud offering in beta mode about one year after HP did with general availability expected later this year. Amazon is famous for it’s low price structure and it will be a challenge for HP and IBM to compete.

Mirantis, a 13-year old company with a history of strong professional services has immersed itself into OpenStack consulting and training in the last couple of years. They help service providers and enterprises build and deploy OpenStack cloud infrastructure. Some of their customers include NASA, Dell, AT&T, and Cisco. Another customer, Internap, launched its OpenStack-based public cloud service back in October 2011 and was assisted by Mirantis. I installed OpenStack in a personal lab at home and encountered several obstacles along the way. It is not easy to set up. For simple lab installations, DevStack is a good option. However, for production deployments, there is a lot of complexity that needs to be understood and overcome for enterprises to deploy clouds. Companies like Mirantis help bridge that gap.


OpenStack Lab Installation with DevStack

I have been experimenting with OpenStack in a personal lab at home. My motivation has been to get some hands on exposure with Quantum. OpenStack is definitely not a trivial task to set up. I have built it with DevStack, a script that builds complete OpenStack development environments, and have encountered a few bumps along the road. The first one was of package dependencies. I have learned that unlike on our personal laptops running MacOS or Windows, when running a Linux server, such as Ubuntu, patching the kernel to the latest release is not always a good idea. In the case of DevStack, I ran into package dependency errors, which are a nightmare to resolve. There is no general consensus on forums for how to mitigate them.

The only way I was able to get around this was to run the DevStack script with an unaltered kernel from Ubuntu 12.04 LTS. That means no running of ‘apt-get update’ followed by ‘apt-get upgrade’. I was able to get OpenStack successfully running with Kernel 3.2.0-29, which is what Ubuntu 12.04 LTS comes with natively. I’ve attended a few OpenStack meetups and my experience is consistent with those of other attendees who I have interacted with. In hindsight, it is not surprising that DevStack broke because OpenStack has so many constant code changes and moving parts. DevStack, which pulls the latest release from Git Hub, is likely to break if a major variable change, such as kernel upgrade, is introduced.

I got some help from the inimitable Brent Salisbury, whose blog posts have come in handy on several occasions. Next I plan to add customizations to my OpenStack installation, such as adding Quantum plug-ins rather than using nova-network.

South Bay OpenStack Meetup – Where are the Networkers?

Just a quick note before the weekend. Yesterday I attended a South Bay OpenStack Meetup organized by Mirantis, a provider of OpenStack services. Over 100 people had RSVP’d for the event, which was held on Yahoo!’s campus. About 50 attended.

The event featured an introductory presentation by Mirantis on the OpenStack architecture and featured excellent coverage on the messaging between the various components and APIs. Co-resenting was Lee Xie, Senior Technical Engagement Manager from Mirantis, who had earlier in the day published a detailed, albeit subjective comparison of OpenStack versus VMware.

What struck me as amazing was the lack of questions and familiarity with Quantum, the networking component of OpenStack that has been out since Folsom was released in September 2012. I had never expected myself to be the only person asking questions about Quantum at an OpenStack event! OpenStack itself has been around since 2010 or so, and it is possible that most of the attendees had server and storage background. JSON, REST, Puppet, and Rabbit were the more fluid topics of discussion. I drew puzzled looks when I broached the subject of Floodlight and OpenFlow.

The meetups are scheduled for every other week with Beginners and Intermediate tracks held at the same time in different rooms. Maybe I’ll attend the Intermediate track next time.

Book Review – OpenStack Cloud Computing Cookbook

In December 2012, I participated in a contest on Scott Lowe’s blog and won a copy of Kevin Jackson’s OpenStack Cloud Computing Cookbook. I’ve been reading it to progress my OpenStack lab and have a few thoughts on it.

The most important thing to keep in mind about this book is that it is not about the design philosophy or goals of OpenStack. If you are looking for an OpenStack 101 book, this is not the one. It is very simply, a cookbook with precise recipes or tasks to set up and manage OpenStack cloud environments, and doesn’t pretend to be anything other than that. Hence, it is very difficult for it to be a meaningful and long-lasting book. This is unlikely to be on my bookshelves in a year or two. OpenStack has been evolving very rapidly and at the time the book was published, in September 2012, OpenStack came out with a new release – Folsom. The book was developed on the Essex platform, which was around April 2012. Folsom is a major release as it contains the Quantum networking component. Consequently, the book only covers Nova Networking (Chapter 10), which supports Flat networking, Flat networking with DHCP, and VLAN Manager. With, Quantum, users are presented a backend platform from which they can leverage plugins to pick network services from many vendors. Similarly, there is no mention of Cinder, the full-blown component that covers block storage. Instead, the book only talks about Nova Volumes (Chapter 8) for block storage support.

The OpenStack components present in Essex receive coverage in the following chapters:

  • Nova (Compute) – Chapters 1 (Starting) and 2 (Administering)
  • Keystone (Identity) – Chapter 3
  • Swift (Storage) – Chapters 4 (Installing), 5 (Using), and 6 (Administering)
  • Glance (Image) – Chapter 7
  • Horizon (Dashboard) – Chapter 9

Chapter 11 discusses how to provision OpenStack in Data Centers and discusses the tools and techniques for automating tasks. The absence of DevStack is notable here.

Chapters 12 and 13 cover Monitoring and Troubleshooting respectively.

Most of the 100-odd tasks are written in a three-tiered Getting ready / How to do it / How it works format. It is not a book you would read from start to finish; instead you would pick tasks that are important to you. However, though there are several screenshots and code snippets, there is not a single diagram, either block or network. This is another major shortcoming in the book. One would expect at least a few of the How it works sections to have diagrams to illustrate, conceptually, how the task was realized, such as the flow of packets between VMs in Chapter 10 (OpenStack Networking).

However, it is a good resource for monitoring and troubleshooting tasks and might serve your needs there. Otherwise, my biggest complaint of OpenStack Cloud Computing Cookbook was that it was largely obsolete on the day it was released.

Open vSwitch 1.4 installation from package on Ubuntu 12.04

In trying to get a more grounded feeling for OpenStack I’ve decided to build a home lab. One step involves configuring Open vSwitch to bridge with VMs. In this post I shall cover the Open vSwitch (OVS) build process along with KVM installation. Future posts shall cover more detailed configurations and scenarios along with videos.

While I am more familiar with the CentOS/RHE flavors of Linux, there seems to be more support for OVS on the Debian/Ubuntu platform. So in this post I am covering Ubuntu 12.04 LTS. There are two ways to install OVS:

  • Use Ubuntu’s apt-get installer to install packages – easier
  • Build from source code – more difficult

This post is aiming at the low-hanging fruit of building from the package. The drawback is that newer features are unavailable in the package. The package version of OVS is 1.4.0. The most stable Long Term release, as of writing, is 1.4.3, while the latest release, 1.7.1, includes support for VXLAN and Open Flow. I plan to document my findings with various builds and Linux flavors in future posts.

As I mentioned, I built OVS 1.4.0 off of Ubuntu 12.04 LTS (Long Term Support), which runs kernel version 3.2. The following steps are taken from various documents on the OVS site, while the outputs are excerpts from my lab.

root@pakdude-02:~# uname -a
Linux pakdude-02 3.2.0-34-generic #53-Ubuntu SMP Thu Nov 15 10:48:16 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
root@pakdude-02:~# apt-get install build-essential fakeroot openvswitch-switch openvswitch-common openvswitch-datapath-source

Keep in mind that additional packages, such as dkms (Dynamic Kernel Module Support), were installed as a result because they were pre-requisites.
The following output is good:

DKMS: build completed.

Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.2.0-34-generic/updates/dkms/

Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.2.0-34-generic/updates/dkms/


DKMS: install completed.
Setting up openvswitch-switch (1.4.0-1ubuntu1.3) ...
 * Inserting openvswitch module
 * /etc/openvswitch/conf.db does not exist
 * Creating empty database /etc/openvswitch/conf.db
 * Starting ovsdb-server
 * Configuring Open vSwitch system IDs
 * Starting ovs-vswitchd
 * Enabling gre with iptables

OVS has now been built. We will verify shortly. But first, we need to install KVM, a full-blown virtualization solution for Linux, and libvirt-bin, a daemon that loads the KVM modules. KVM also inclue virsh, which is a tool to manage (create, start, stop, etc) virtual domains or networks. Remember, KVM requires libvirt-bin.

root@pakdude-02:~# apt-get install libvirt-bin

Note that this will install bridge-utils and ebtables as well. We will get to that shortly. First, we want to destroy the default network created by libvirt-bin, which is virbr0. OVS will supply the network instead.

root@pakdude-02:~# ifconfig virbr0
virbr0    Link encap:Ethernet  HWaddr 4e:c0:0d:41:e3:0c  
          inet addr:  Bcast:  Mask:
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

root@pakdude-02:~# virsh net-destroy default
Network default destroyed

root@pakdude-02:~# virsh net-autostart --disable default
Network default unmarked as autostarted

root@pakdude-02:~# ifconfig virbr0
virbr0: error fetching interface information: Device not found

Now we have to actually install KVM.

root@pakdude-02:~# apt-get install kvm

Some additional packages are installed in the process.
Keep in mind that ebtables is not needed, so remove it. OVS will play the role of the bridge.

root@pakdude-02:~# apt-get purge ebtables

bridge still showed up in lsmod | grep bridge, but there was no need to rmmod it (as shown in many other guides on the web) as it was gone upon the next reboot. Remember, OVS will assume the bridging functionality. Some guides mention Bridge Compatibility installation. However, I do not see the need. Bridge Compatibility provides a way for applications that use the Linux bridge to gradually migrate to OVS. Programs that ordinarily control the Linux bridge module, such as brctl, instead control the OVS kernel-based switch. If you do not already depend on these programs, then you do not need bridge compatibility.

root@pakdude-02:~# service openvswitch-switch status
ovsdb-server is running with pid 1104
ovs-vswitchd is running with pid 1125
root@pakdude-02:~# ovs-vsctl show
    ovs_version: "1.4.0+build0"

We’re good to go. Additionally, these are the relevent processes that are now running:

root@pakdude-02:~# ps -face | grep ovs
root      1103     1 TS   29 23:45 ?        00:00:00 ovsdb-server: monitoring pid 1104 (healthy)                                                                                                                                                                                                                                                                                                                                                                       
root      1104  1103 TS   29 23:45 ?        00:00:00 ovsdb-server /etc/openvswitch/conf.db -vANY:CONSOLE:EMER -vANY:SYSLOG:ERR -vANY:FILE:INFO --remote=punix:/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,manager_options --private-key=db:SSL,private_key --certificate=db:SSL,certificate --bootstrap-ca-cert=db:SSL,ca_cert --no-chdir --log-file=/var/log/openvswitch/ovsdb-server.log --pidfile=/var/run/openvswitch/ --detach --monitor
root      1124     1 TS   29 23:45 ?        00:00:00 ovs-vswitchd: monitoring pid 1125 (healthy)                                                                                                                                                                                                 
root      1125  1124 TS   29 23:45 ?        00:00:00 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vANY:CONSOLE:EMER -vANY:SYSLOG:ERR -vANY:FILE:INFO --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ --detach --monitor
root      2346  2183 TS   19 23:57 pts/1    00:00:00 grep --color=auto ovs

And that’s about it. Hopefully I’ll get some functionality and configurations up here soon.