In order to prepare for my CCIE R&S recert, and in an attempt to save trees, I bought the e-book (Kindle edition) of CCIE Routing and Switching Certification Guide, 4th Edition from Amazon.com. The print edition costs at least $20 more than the Kindle edition. However, after a few weeks of reading it on my first generation Kindle, I returned it to Amazon and bought the print edition instead. Here are the reasons why:
Last month I wrote how I built Open vSwitch 1.4.0 package on Ubuntu 12.04. Immediately afterwards I left my lab and when I returned to it, nearly a month later, I ran an apt-get upgrade out of habit. Consequently, the kernel got upgraded from 3.2.0-34 to 3.2.0-36 and I ran into the following error when starting the OVS service:
root@pakdude-02:~# uname -a Linux pakdude-02 3.2.0-36-generic #57-Ubuntu SMP Tue Jan 8 21:44:52 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux root@pakdude-02:~# service openvswitch-switch start FATAL: Module openvswitch_mod not found. * Inserting openvswitch module Module has probably not been built for this kernel. For instructions, read /usr/share/doc/openvswitch-datapath-source/README.Debian FATAL: Module openvswitch_mod not found. * Inserting openvswitch module root@pakdude-02:~#
It turns out that this was because the kernel modules also needed to be updated. The following command did the trick:
root@pakdude-02:~# module-assistant auto-install openvswitch-datapath
I could have simply typed the following instead with the same consequences.
root@pakdude-02:~# m-a a-i openvswitch-datapath
As per the documentation, module-assistant aims to facilitate the process of building kernel modules from source. In other words, this needs to be run each time the kernel is upgraded.
In December 2012, I participated in a contest on Scott Lowe’s blog and won a copy of Kevin Jackson’s OpenStack Cloud Computing Cookbook. I’ve been reading it to progress my OpenStack lab and have a few thoughts on it.
The most important thing to keep in mind about this book is that it is not about the design philosophy or goals of OpenStack. If you are looking for an OpenStack 101 book, this is not the one. It is very simply, a cookbook with precise recipes or tasks to set up and manage OpenStack cloud environments, and doesn’t pretend to be anything other than that. Hence, it is very difficult for it to be a meaningful and long-lasting book. This is unlikely to be on my bookshelves in a year or two. OpenStack has been evolving very rapidly and at the time the book was published, in September 2012, OpenStack came out with a new release – Folsom. The book was developed on the Essex platform, which was around April 2012. Folsom is a major release as it contains the Quantum networking component. Consequently, the book only covers Nova Networking (Chapter 10), which supports Flat networking, Flat networking with DHCP, and VLAN Manager. With, Quantum, users are presented a backend platform from which they can leverage plugins to pick network services from many vendors. Similarly, there is no mention of Cinder, the full-blown component that covers block storage. Instead, the book only talks about Nova Volumes (Chapter 8) for block storage support.
The OpenStack components present in Essex receive coverage in the following chapters:
- Nova (Compute) – Chapters 1 (Starting) and 2 (Administering)
- Keystone (Identity) – Chapter 3
- Swift (Storage) – Chapters 4 (Installing), 5 (Using), and 6 (Administering)
- Glance (Image) – Chapter 7
- Horizon (Dashboard) – Chapter 9
Chapter 11 discusses how to provision OpenStack in Data Centers and discusses the tools and techniques for automating tasks. The absence of DevStack is notable here.
Chapters 12 and 13 cover Monitoring and Troubleshooting respectively.
Most of the 100-odd tasks are written in a three-tiered Getting ready / How to do it / How it works format. It is not a book you would read from start to finish; instead you would pick tasks that are important to you. However, though there are several screenshots and code snippets, there is not a single diagram, either block or network. This is another major shortcoming in the book. One would expect at least a few of the How it works sections to have diagrams to illustrate, conceptually, how the task was realized, such as the flow of packets between VMs in Chapter 10 (OpenStack Networking).
However, it is a good resource for monitoring and troubleshooting tasks and might serve your needs there. Otherwise, my biggest complaint of OpenStack Cloud Computing Cookbook was that it was largely obsolete on the day it was released.
We are two weeks into 2013, so these goals might appear a bit late, but are better that way than never being made. I’m limiting these goals to the professional arena as most of the readers probably won’t care whether I learned French, took 30 wickets in my club’s summer cricket league, or learned how to make the perfect cappuccino.
With that said, here are the professional goals that I will strive to achieve in 2013:
- Immerse myself in OpenStack and its Quantum component. Specifically, I want to build a cloud in my home lab and create complicated scenarios, including, but not limited to:
- Create virtual Data Centers
- Easily spin off VMs and move them across L2 and L3 domains
- Build and configure Open vSwitch and a Floodlight OpenFlow controller to achieve the tasks above
- Document my findings and release them to the public in easy-to-understand videos and screencasts.
- Watch all of Ivan Pepeljnak’s webinars. So far I’ve only watched about a third.
- Attain a working knowledge of Python via Codeacademy.
- Recertify my CCIE status.
- Play a major role in building a product.
I’ll revisit in about a year to see how I fared!
I was in Karachi, Pakistan recently for brief visit during the Christmas holidays. Though the purpose of my visit was personal, I did manage to squeeze in some time speaking to professionals who are intimate with the state of networking in Pakistan. In particular I spoke with one individual at Cisco Pakistan who did not wish to be named, but is very familiar with the largest networks in Pakistan.
First, a brief word about telecommunications. Mobile networks in Pakistan currently utilize GPRS and EDGE technologies. Plans to roll out 3G and pseudo 4G technologies have been put on hold. However, broadband speeds to homes and offices have improved significantly over the years, with WiMAX deployments common in Karachi.
Amongst the various ISPs in Pakistan, the biggest player by far is PTCL, which, as of 2006, is a semi-private corporation. PTCL is a Cisco shop that is investing heavily in L2 and L3 MPLS backbones for their customers. For many years, up into the mid-2000s, VSAT communications and dialup were the only means of Internet connectivity. So it was refreshing to see this step being taken.
High Availability is a tough ask in Pakistan with very few enterprises deploying redundant links or nodes. The exceptions are the larger banks. Generally speaking, it has been difficult to educate CIOs in Pakistan on the need for high availability. Likewise, structured cabling and cooling in Data Centers is often neglected or simply misunderstood.
On the technology front, the biggest banks in Pakistan are among the few to deploy Nexus 7Ks. Service Providers such as PTCL deploy CRS’. Virtualization has also yet to make a significant penetration in Pakistan Data Centers. Only the largest banks carry ESX licenses. My questions about SDN, overlay networks, and private clouds met bemused expressions.
However, the country has no shortage of talent. The number of universities that offer degrees in computer science and computer engineering has increased significantly since the early 1990s. The past 20 years has seen some brilliant professionals rise from Hamdard Institute of Information Technology (HIIT), Usman Institute of Technology (UIT), Ghulam Ishaq Khan Institiute of Engineering Sciences and Technology (GIK), and the School of Electrical Engineering and Computer Sciences (SEECS) at the National University of Science and Technology (NUST). R&D work in engineering sciences, unlike that in the natural sciences, is of a much higher quality and comparable to any international institute.
Take, for example, a startup incubated out of NUST, called xFlow Research, which is doing fantastic work in porting Open vSwitch to the Marvell xCat and LION platforms. On the Open vSwitch mailing list archives, about 10% of the contributions come from Pakistanis.
Clearly, despite all the challenges that Pakistani enterprises face with proprietary offerings from pure-play networking vendors and a politically unstable environment, the open source world offers a lot of potential to Pakistan networking industry. I wouldn’t be surprised if 2013 saw some major contributions to OpenStack being made by Pakistani companies.
In trying to get a more grounded feeling for OpenStack I’ve decided to build a home lab. One step involves configuring Open vSwitch to bridge with VMs. In this post I shall cover the Open vSwitch (OVS) build process along with KVM installation. Future posts shall cover more detailed configurations and scenarios along with videos.
While I am more familiar with the CentOS/RHE flavors of Linux, there seems to be more support for OVS on the Debian/Ubuntu platform. So in this post I am covering Ubuntu 12.04 LTS. There are two ways to install OVS:
- Use Ubuntu’s apt-get installer to install packages – easier
- Build from source code – more difficult
This post is aiming at the low-hanging fruit of building from the package. The drawback is that newer features are unavailable in the package. The package version of OVS is 1.4.0. The most stable Long Term release, as of writing, is 1.4.3, while the latest release, 1.7.1, includes support for VXLAN and Open Flow. I plan to document my findings with various builds and Linux flavors in future posts.
As I mentioned, I built OVS 1.4.0 off of Ubuntu 12.04 LTS (Long Term Support), which runs kernel version 3.2. The following steps are taken from various documents on the OVS site, while the outputs are excerpts from my lab.
root@pakdude-02:~# uname -a Linux pakdude-02 3.2.0-34-generic #53-Ubuntu SMP Thu Nov 15 10:48:16 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux root@pakdude-02:~# apt-get install build-essential fakeroot openvswitch-switch openvswitch-common openvswitch-datapath-source
Keep in mind that additional packages, such as dkms (Dynamic Kernel Module Support), were installed as a result because they were pre-requisites.
The following output is good:
DKMS: build completed. openvswitch_mod: Running module version sanity check. - Original module - No original module exists within this kernel - Installation - Installing to /lib/modules/3.2.0-34-generic/updates/dkms/ brcompat_mod.ko: Running module version sanity check. - Original module - No original module exists within this kernel - Installation - Installing to /lib/modules/3.2.0-34-generic/updates/dkms/ depmod.... DKMS: install completed. Setting up openvswitch-switch (1.4.0-1ubuntu1.3) ... * Inserting openvswitch module * /etc/openvswitch/conf.db does not exist * Creating empty database /etc/openvswitch/conf.db * Starting ovsdb-server * Configuring Open vSwitch system IDs * Starting ovs-vswitchd * Enabling gre with iptables
OVS has now been built. We will verify shortly. But first, we need to install KVM, a full-blown virtualization solution for Linux, and libvirt-bin, a daemon that loads the KVM modules. KVM also inclue virsh, which is a tool to manage (create, start, stop, etc) virtual domains or networks. Remember, KVM requires libvirt-bin.
root@pakdude-02:~# apt-get install libvirt-bin
Note that this will install bridge-utils and ebtables as well. We will get to that shortly. First, we want to destroy the default network created by libvirt-bin, which is virbr0. OVS will supply the network instead.
root@pakdude-02:~# ifconfig virbr0 virbr0 Link encap:Ethernet HWaddr 4e:c0:0d:41:e3:0c inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) root@pakdude-02:~# virsh net-destroy default Network default destroyed root@pakdude-02:~# virsh net-autostart --disable default Network default unmarked as autostarted root@pakdude-02:~# ifconfig virbr0 virbr0: error fetching interface information: Device not found
Now we have to actually install KVM.
root@pakdude-02:~# apt-get install kvm
Some additional packages are installed in the process.
Keep in mind that ebtables is not needed, so remove it. OVS will play the role of the bridge.
root@pakdude-02:~# apt-get purge ebtables
bridge still showed up in lsmod | grep bridge, but there was no need to rmmod it (as shown in many other guides on the web) as it was gone upon the next reboot. Remember, OVS will assume the bridging functionality. Some guides mention Bridge Compatibility installation. However, I do not see the need. Bridge Compatibility provides a way for applications that use the Linux bridge to gradually migrate to OVS. Programs that ordinarily control the Linux bridge module, such as brctl, instead control the OVS kernel-based switch. If you do not already depend on these programs, then you do not need bridge compatibility.
root@pakdude-02:~# service openvswitch-switch status ovsdb-server is running with pid 1104 ovs-vswitchd is running with pid 1125 root@pakdude-02:~# ovs-vsctl show ab15a0d5-7c66-4388-b921-5d4397a7608b ovs_version: "1.4.0+build0"
We’re good to go. Additionally, these are the relevent processes that are now running:
root@pakdude-02:~# ps -face | grep ovs root 1103 1 TS 29 23:45 ? 00:00:00 ovsdb-server: monitoring pid 1104 (healthy) root 1104 1103 TS 29 23:45 ? 00:00:00 ovsdb-server /etc/openvswitch/conf.db -vANY:CONSOLE:EMER -vANY:SYSLOG:ERR -vANY:FILE:INFO --remote=punix:/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,manager_options --private-key=db:SSL,private_key --certificate=db:SSL,certificate --bootstrap-ca-cert=db:SSL,ca_cert --no-chdir --log-file=/var/log/openvswitch/ovsdb-server.log --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor root 1124 1 TS 29 23:45 ? 00:00:00 ovs-vswitchd: monitoring pid 1125 (healthy) root 1125 1124 TS 29 23:45 ? 00:00:00 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vANY:CONSOLE:EMER -vANY:SYSLOG:ERR -vANY:FILE:INFO --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor root 2346 2183 TS 19 23:57 pts/1 00:00:00 grep --color=auto ovs root@pakdude-02:~#
And that’s about it. Hopefully I’ll get some functionality and configurations up here soon.
This week, the Lean Startup conference was held in San Francisco. The Lean Startup philosophy borrows its roots from the lean management method of manufacturing, with Kanban or Just-in-time processing at the center of the design principle. It basically advocates startups to minimize outside funding, not strive for the perfect product (think Minimal Viable Product), be flexible (think Pivot), and cater the product completely to the customer’s needs, all with the goal of being a highly efficient company.
Not all startups can be Lean Startups
Indeed, some startups cannot afford to employ a Pivot. Infrastructure or hardware companies come to mind, especially when they’ve already taken $50 million of investment. This is what Plexxi has done without a complete solution to show for yet.
I was listening to the recent Packet Pushers show #126 sponsored by Plexxi. While their approach is a creative one, I’m not sure whether it is viable. In a nutshell, Plexxi brings optical technology, in the form of WDM, to the Data Center and flattens traditional hierarchical network designs. When I first started learning about network designs, the classical approach was the 3-tier Core-Distribution-Access model. In the mid-2000s this got reduced to a Collapsed Core. What Plexxi proposes is a flat topology, eliminating the need for Core switches in the Data Center.
Plexxi adopted the SDN approach of a programmable controller (a virtual appliance) that pushes policies to its switches. The policies are intended to optimize data path flow for affinitized traffic. Applications that are more sensitive of certain resources are classified in Affinity Networks. Some example of the constraints or sensitivities that Plexxi’s Director of Product Management, Marten Terpstra, described include:
Plexxi switches use merchant silicon (Broadcom ASICs) to form an Ethernet ring on top of a WDM lightwave. By changing lambdas, Layer-1 connections between switches can be changed according to the application requirements.
Plexxi uses their own closed APIs for communication between their switch interfaces and their controller, in order to convey their message of affinities. However, they open up their proprietary northbound API for user-to-controller communication so users can write scripts, for example, by using REST APIs. Interestingly, they are a member of Open Network Foundation. The Controller places TCAM entries in switches based on application requirements for affinitized traffic.
Terpstra discussed two use cases:
- Affinitized iSCSI traffic for most bandwidth with least number of hops
- Cloud provider – Use a Plexxi ring as a premium service to affinitize traffic.
In neither case are the results mentioned.
Okay, so so far Plexxi’s solution is a 1 RU box that can prioritize traffic based on hop-count and bandwidth. I fail to see much of a business case there. Any network engineer worth his or her salt will tell you that there is more to traffic classification and prioritization than just hop-count and bandwidth. Financial trading institutions would be more concerned about latency guarantees. Hop Count alone is a flimsy criterion to classify important traffic, regardless of whether a cute term like Affinity Network is given to that classification. High Availability is a critical issue that a ring topology exacerbates. As Doug Gourlay of Arista mentions, unnecessary downtime is introduced any time you add new nodes because the ring is broken. Moreover, the network is reduced to a split brain model in the even of just two nodes going down. Depending on the Controller placement, this could have adverse outcomes. The thing about outages is that we can never control where they occur. Gourlay rightly puts it:
I thought Token Ring died for good reasons… why is someone trying to bring it back?
Getting back to the Lean Startup idea, Terpstra said “Our Layer-3 affinities are coming”. Plexxi is targeting Christmas 2012 for 1.0 version of Layer-3 capabilities. Until then Plexxi only has a Layer-2 switch with no quotable value to show for $50 million in investment. Not a good time to Pivot.
Reports of the death of the Core switch in the Data Center have been greatly exaggerated.