Avoiding Shellshock in Mac OSX

September 26, 2014 Umair Hoodbhoy Leave a comment

Shellshock is a vulnerability in bash (the shell that comes with Mac OSX) that surfaced in late September 2014 and has the potential to do more harm than Heartbleed that made headlines in April 2014. Apple ships OSX with an old version of bash. According to this site, Shellshock can potentially be used to execute arbitrary code on environment variables that are passed to child processes. What follows is my approach to hardening my Macbook.

You know you are vulnerable in OSX if you run the following at the Terminal window prompt:

hoodbu@pakdude-mbp /~ (499) env x='() { :;}; echo vulnerable' bash -c "echo this is a test"
vulnerable
this is a test
hoodbu@pakdude-mbp /~ (500)

This is because of the version of bash that I had on my Macbook:

hoodbu@pakdude-mbp /~ (501) bash --version
GNU bash, version 3.2.51(1)-release (x86_64-apple-darwin13)
Copyright (C) 2007 Free Software Foundation, Inc.
hoodbu@pakdude-mbp /~ (502)

Following the instructions given at Stack Exchange, I ran the following:

hoodbu@pakdude-mbp /~ (528) mkdir bash-fix
hoodbu@pakdude-mbp /~ (529) cd bash-fix/
hoodbu@pakdude-mbp /bash-fix (530) curl https://opensource.apple.com/tarballs/bash/bash-92.tar.gz | tar zxf -
-bash: /sw/bin/tar: Bad CPU type in executable
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0 4088k    0 16384    0     0   4927      0  0:14:09  0:00:03  0:14:06  4927
curl: (23) Failed writing body (0 != 16384)
hoodbu@pakdude-mbp /bash-fix (531)

This error was because of my version of ‘tar’. Somehow, my ‘/sw/bin/tar’ is a PowerPC-only binary of tar probably because I once owned a PowerPC-based Mac and after upgrading many years ago my version of ‘tar’ somehow didn’t get updated.

hoodbu@pakdude-mbp /bash-fix (534) /usr/bin/tar --version
bsdtar 2.8.3 - libarchive 2.8.3
hoodbu@pakdude-mbp /bash-fix (535) tar --version
-bash: /sw/bin/tar: Bad CPU type in executable
hoodbu@pakdude-mbp /bash-fix (536) which tar
/sw/bin/tar

So I just used ‘/usr/bin/tar’ and will deal with ‘/sw/bin/tar’ later. Moving on,

hoodbu@pakdude-mbp /bash-fix (537) curl https://opensource.apple.com/tarballs/bash/bash-92.tar.gz | /usr/bin/tar zxf -
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 4088k  100 4088k    0     0   603k      0  0:00:06  0:00:06 --:--:--  607k
hoodbu@pakdude-mbp /bash-fix (539) cd bash-92/bash-3.2
hoodbu@pakdude-mbp /bash-3.2 (540) curl https://ftp.gnu.org/pub/gnu/bash/bash-3.2-patches/bash32-052 | patch -p0
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3250  100  3250    0     0   2041      0  0:00:01  0:00:01 --:--:--  2042
patching file builtins/common.h
patching file builtins/evalstring.c
patching file variables.c
patching file patchlevel.h
hoodbu@pakdude-mbp /bash-3.2 (541) curl http://alblue.bandlem.com/bash32-053.patch | patch -p0
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1118  100  1118    0     0    803      0  0:00:01  0:00:01 --:--:--   803
patching file parse.y
patching file patchlevel.h
hoodbu@pakdude-mbp /bash-3.2 (542) cd ..
hoodbu@pakdude-mbp /bash-92 (543) xcodebuild
xcode-select: note: no developer tools were found at '/Applications/Xcode.app', requesting install. Choose an option in the dialog to download the command line developer tools.

Apparently I had had ‘xcodebuild’, but not the way Apple wants it. So I installed it from the App Store. At 2.46 GB, it took a while to download, but once installing, running as sudo, and agreeing to the EULA, the rest was straightforward:

hoodbu@pakdude-mbp /bash-92 (544) xcodebuild
Agreeing to the Xcode/iOS license requires admin privileges, please re-run as root via sudo.
hoodbu@pakdude-mbp /bash-92 (545) sudo xcodebuild
Password:
You have not agreed to the Xcode license agreements. You must agree to both license agreements below in order to use Xcode.
Hit the Enter key to view the license agreements at '/Applications/Xcode.app/Contents/Resources/English.lproj/License.rtf'
<long EULA skipped>
hoodbu@pakdude-mbp /bash-92 (547) sudo xcodebuild
<long output skipped>
** BUILD SUCCEEDED **
hoodbu@pakdude-mbp /bash-92 (548) sudo cp /bin/bash /bin/bash.old
hoodbu@pakdude-mbp /bash-92 (549) sudo cp /bin/sh /bin/sh.old
hoodbu@pakdude-mbp /bash-92 (550) build/Release/bash --version # GNU bash, version 3.2.53(1)-release
GNU bash, version 3.2.53(1)-release (x86_64-apple-darwin13)
Copyright (C) 2007 Free Software Foundation, Inc.
hoodbu@pakdude-mbp /bash-92 (551) build/Release/sh --version   # GNU bash, version 3.2.53(1)-release
GNU bash, version 3.2.53(1)-release (x86_64-apple-darwin13)
Copyright (C) 2007 Free Software Foundation, Inc.
hoodbu@pakdude-mbp /bash-92 (552) sudo cp build/Release/bash /bin
hoodbu@pakdude-mbp /bash-92 (553) sudo cp build/Release/sh /bin
hoodbu@pakdude-mbp /bash-92 (554) bash --version
GNU bash, version 3.2.53(1)-release (x86_64-apple-darwin13)
Copyright (C) 2007 Free Software Foundation, Inc.
hoodbu@pakdude-mbp /bash-92 (555)

Finally, this is the indicator that my Macbook is no longer vulnerable to Shellshock:

hoodbu@pakdude-mbp /bash-92 (555) env x='() { :;}; echo vulnerable' bash -c 'echo hello'
bash: warning: x: ignoring function definition attempt
bash: error importing function definition for `x'
hello
hoodbu@pakdude-mbp /bash-92 (556)

I hope you find this useful.

Ethernet, Speculation

IDF14 – Will Bare Metal Servers Obviate Bare Metal Switches?

September 23, 2014 Umair Hoodbhoy 1 Comment

Recently I wrote about the Networking Field Day 8 presentations on Nuage Networks and Big Switch Networks. A noticeable absentee at Networking Field Day 8 was the co-host of the popular Packet Pushers show, Greg Ferro. What was so important that kept Mr. Ferro away from NFD8? Well, it turns out that he was attending Intel Developer Forum 2014 and discussed his findings on his show – The Network Break (I guess you can call me a Greg Ferro stalker). This prompted me to dig a bit deeper into Intel’s Software Defined Infrastructure vision and what I think it means to the networking industry.

Intel’s announcements included new products such as the XL710 controller and E5 chipset, and technologies, such as QuickAssist Network Acceleration APIs and Data Plane Development Kit (DPDK).

DPDK has actually been around since 2010. As defined on its website, it is a set of libraries and drivers for fast packet processing on x86 platforms. It runs mostly in Linux userland. This allows for higher levels of packet processing throughput than what is achievable using the standard Linux kernel network stack. In fact, according to these slides, it can achieve a 25X improvement in per core L3 packet performance over standard Linux. Using DPDK, the latest Intel chips can support Geneve, which is a highly extensible UDP encapsulation for overlays. Geneve claims to perform flexible packet matching of any type of tunnel protocol (such as VXLAN and NVGRE). Within the Geneve Header is an Options field that can contain metadata and context, which is invaluable for NFV & service chaining). So, it is not surprising that Intel has a partnership with VMware (the champions of overlay networks) that is catered to NFV solutions.

The Ethernet Controller XL710 is 40 Gbps ready on a single virtual core, 160 Gbps per CPU socket. It can terminate Geneve tunnels at line rate (39.39 Gbps on the 40 Gbps adapters) as the IDF14 demos showed. The reason it can do this is because of Receive Side Scaling for VXLAN, which balances CPU utilization across cores.

Some of the packet/security/compression acceleration features of the E5-2600 v3 chipsets, powered by QuickAssist technology (which does stateless offload and protocol acceleration), are 100 Gbps SSL Termination (a boon for SEO), 160 Kops (key operations for IPSec), 80 Gbps Platform compression (applicable for Big Data analytics like Hadoop), which should keep pure play networking vendors (including firewall, VPN Concentrators, and load balancer) on their toes.

The overlay vs underlay network debate has become a hot topic in recent years, perhaps best exemplified by the Cisco ACI vs VMware NSX solutions. VMware believes that overlays on top of bare metal servers running X86 chips are the way of the future. They believe that protocol offload technologies like QuickAssist are the solution for building scalable infrastructures. Pure play vendors like Cisco believe that there is still value in custom networking ASICs on switches that form the underlays. Still other networking startup vendors like Pica8, Cumulus Networks, and Big Switch Networks are the poster children of bare metal switches, i.e. switches that leverage merchant silicon, such as Broadcom or Marvell, and whose sheet metal is assembled by white box vendors such as Celestica, Delta Networks, Acton, or Quanta.

How will Intel’s recent announcements affect networking vendors? Well, network virtualization poses very different challenges from server virtualization. Protocol offload has been around for several years, but isn’t as ubiquitous as you’d think: I ran into performance issues first hand with TCP offload in 2011 when disabling it used to give much better results. And simply slapping an overlay on top doesn’t solve every networking problem. Scaling in network virtualization is far more difficult than scaling in server virtualization. For example, the number of ACLs needed grows quadratically as the number of web servers or database servers increases linearly. I think the future is still bright for bare metal switch vendors, but I would love to hear back from you.

Networking Field Day, SDN, Startups

Deconstructing Big Switch Networks at NFD8

September 19, 2014 Umair Hoodbhoy 1 Comment

I recently caught up with the presentation made by Big Switch Networks at Networking Field Day 8.

Founder Kyle Forster kicked things off with an introduction to Big Switch. He used the term ‘Hyperscale Data Centers’ to describe the data center of today and tomorrow that Big Switch targets. Big Switch has two products based on the following three principles:

Bare metal switches for Hardware using Broadcom ASICs.
Controller-based design for software
Core-pod architecture replacing the traditional Core-Aggregation-Edge tier.

The two products are:

Big Tap Monitoring Fabric – Taps for offline network monitoring
Big Cloud Fabric – Clos switching fabric

Next up, CTO Rob Sherwood went into more detail. He defined the core-pod architecture as essentially a spine-leaf architecture where there are 16 pods (racks that have servers), at the top of which are two leaves. Each server is dual-connected to a leaf via 10G links. The leaves themselves connect up to spines via 40G links. The leaf switches are 48x10G and 6x40G for uplinks; the spine switches are 32x40G. So the maximum number of spine switches in a pod is 6. (In a leaf-spine fabric every leaf must connect to every spine). That also means a maximum of 32 leaves can connect to a spine. These numbers will definitely increase in future generations of switches when Broadcom can produce them at scale. This solution is targeted at Fortune 1000 companies, not really as much on smaller enterprises. Pods are very modular and can be replaced without disrupting the older network designs.

What I thought was pretty cool was the use of Open Network Installation Environment (ONIE) for provisioning. The switches get shipped to customers from Dell or Accton with a very lightweight OS, then as it turn on the box it netboots from the Controller (an ONIE server). Both Switch Light (which is the Big Switch OS), as well as the relevant configurations, get downloaded from the Controller to the switch. LLDP is used to auto-discover the links in the topology, and management software will tell if there are missing or double-connected links.

In the first demo, the Automated Fabric Visibility Tool was used to first allocate and assign roles in the topology. At that point, any errors in cabling would appear in the GUI, which was pretty user-friendly. The Big Cloud Fabric architecture has a dedicated OOB control/management network that connects to the Controller. Amongst the management primitives are a logical L2 segment (ala VLAN) that have logical ports and end-points, tenants that are logical grouping of L2/L3 networks, and logical routers that are the tenant routers for inter-segment or intra-segment routing. Each logical router corresponds to a VRF. VLAN tags can be mixed and matched and added into bridged domains. The use case would be analogous to a multi-tenant environment in each ESX instance, when you declare egress VLAN tags on vswitch in VMware deployments. You have the choice of specifying the tag as global fabric or local to the vswitch. Interestingly, Big Switch used to have an Overlay product two years ago and ended up tossing it away (because they feel they are L2 solutions only, not L3 solutions) to come up with the current solution because they believe it uses the hardware the way it was designed to be used.

The next demo was to create tenants, assign servers and VMs to logical segments by VLAN, physical ports, or port-groups to meet a use case of a common two-tier application.

The fabric in Big Cloud Fabric is analogous to a sheet metal chassis-based fabric that has fabric backplanes, line cards, and supervisors/management modules in that the spine switches are the backplanes, the leaf switches are the line cards, and the Controllers are the supervisors. The analogies actually don’t end with the components. Sherwood explained that traditional chassis switch vendors use proprietary protocols between their backplanes and their line cards that is actually Ethernet and is, therefore, no different from the OOB management network between spine switches and leaf switches. The control planes and the data planes in Big Cloud Fabric are completely decoupled so that in the event of the management switch completely going down, you only lose the ability to change and manage the network. So for example, if a new server comes up, routes for that host don’t get propagated. Of course, if both supervisors in a Nexus 7K go down, the whole switch stops working. If both Controllers go down simultaneously, the time needed to bring up a third Controller is about 5 minutes.

Big Cloud Fabric is based on OpenFlow with extensions. The white box switches that Big Switch sells have Broadcom ASICs that have several types of forwarding tables (programmable memory that can be populated). Some of the early implementations of OpenFlow only exposed the ACL table (which had only about 2000 entries), which didn’t scale well. The way Big Switch implements OpenFlow in Switch Light OS is to expose an L2 table and an L3 table, each with over 100,000 entries. They couldn’t go into more details as they were under NDA with Broadcom. Switch Light OS is Big Switch’s Indigo OpenFlow Agent running on Open Network Linux on x86 or ASIC-based hardware. Whereas traditional networks have clear L2/L3 boundaries in terms of devices, in Big Cloud Fabric L3 packets are routed on the first hop switch. If a tenant needs to talk to another tenant, packets go through a system router, which resides only on a spine switch.

Next up was Service Chaining and Network Functions Virtualization support. Service Chaining is implemented via next-hop forwarding. For example, at a policy level, if one VM or app needed to talk to another app, it could be passed through a service such as a load balancer or firewall (while leveraging the ECMP bits of the hardware) before reaching the destination. The demo showed how to create a policy and then, with a firewall service example, how to apply that policy, which is known as service insertion to an ECMP group. However, it is early days for this NFV use case and for more elaborate needs such as health monitoring, the recommendation is to use OpenStack. Interestingly, Big Switch integrates with OpenStack, but not VMware at this time (it is on the roadmap though).

Operational Simplicity was next on the agenda. Here Big Switch kicked off with the choice of control plane APIs to the user – CLI, GUI, or REST, which, generally speaking, would appeal to network engineers, vCenter administrators, and DevOps personnel respectively. High Availability is designed so that reactions to outages are localized as much as possible. For example, the loss of a spine only reduces capacity, the loss of a leaf is covered by the other leaf in the same rack (thanks to a dedicated link between the two) that has connections to the same servers (so the servers failover to the other leaf via LACP). The Hitless Upgrade process is truly hitless from an application perspective (a few milliseconds of data packets are lost) though capacity is reduced. A feature called Test Path shows the logical (at a policy level) as well as physical path a host takes to reach another host.

The final session was on the Network Monitoring features of Big Switch, namely Big Tap Monitoring Fabric. Sunit Chauhan, head of Product Marketing, said that the monitoring infrastructure is developed using the same bare metal switches that is managed from the same centralized controller. The goal is to monitor and tap every rack and ideally every vswitch. In a campus network that means the ability to filter traffic from all locations to the tools. The Big Tap Monitoring Controller is separate from the Big Cloud Fabric Controller and runs Switch Light OS as well. The example he gave was of a large mobile operator in Japan that needed thousands of taps. The only scalable (in terms of cost and performance) solution to to monitor such a network was to use bare metal Ethernet switches that report to a centralized SDN Controller.

The Big Tap Monitoring demo was based off a common design with a production network (which could be Big Cloud Fabric or traditional L2/L3 networks) with filter ports connected to a Big Tap Controller, which was then connected via delivery ports to the Tool Farm, where all the visibility tools existed.Of course Big Switch eats its own dogfood like every noble startup by deploying Big Tap Monitoring Fabric in its own office. They were able to capture the actual video stream of the NFD event that went out to the Internet from their office. Remote Data Center Monitoring is also supported now (though not demonstrable at NFD8), which reminded me of RSPAN except that this used L2-GRE tunnels.

A few afterthoughts: Big Switch used the marketing term ‘hyperscale data center’ like it was an industry standard and they gave examples of networks that were not hyperscale without explaining how they weren’t. In fact there was a slide that was dedicated to terminology used in a demo, but ‘hyperscale’ was not there. It reminded me of my days in a startup that used that same term in its marketing headlines without ever defining it.

From a personal perspective, in 2010 I worked as a Network Engineer in a large financial exchange where the Big Tap Monitoring Fabric would have been invaluable. Any time a trade was delayed by a couple of seconds resulted in potentially millions of dollars. The exchange would be spared that penalty if it could be proved that the delay was due to the application or the remote network and not the exchange’s own network. At that time we used network monitoring switches to determine where the delay occurred. But the location of those taps was critical. Moreover, it was just not scalable to have taps at every location off of every port. Since it was a reactive (troubleshooting) effort, it was really a Whac-a-Mole exercise. Ultimately, we went with a vendor that built the infrastructure to collect latency data from exchanges, and then offered the results to firms to allow them to monitor data and order execution latency on those markets. But it was expensive and those investments were between $10 and $15 million and ultimately that vendor went out of business. A solution like Big Tap Monitoring Fabric would have been a godsend. If Big Switch can figure out how to keep their costs down, they may have a huge opportunity in hand.

Tech Field Day events are intended to be very technical and this was no different. Slides with Gartner Magic Quadrants are usually met with rolling eyeballs, but I think Big Switch can be forgiven for having one reference to an industry analyst. Apparently according to Dell ‘Oro, in 2013 more ports were shipped from bare metal switches (from vendors such as Dell, Accton, and Quanta) that from Arista, Juniper, and Extreme combined!

While Big Cloud Fabric competes against the Cisco Nexus 7K product line, Big Tap Monitoring goes head to head against Gigamon. It was very refreshing to see a startup take on two behemoths, sheerly with clever engineering and nimble principles.

OpenStack

Live Blog of OpenStack Silicon Valley 2014 Community Event live blog Part 2

September 17, 2014 Umair Hoodbhoy Leave a comment

I attended a variety of track sessions in the afternoon of the OpenStack Silicon Valley Community Event. Part 1 can be viewed here.

From the Networking track:

Popular Options for SDN Deployment in OpenStack

Jonathan LaCour | Dreamhost
Peter Balland | VMWare
Mike Dvorkin | Cisco
Azmir Mohamed | PLUMgrid

Mohamed: PLUMgrid’s baseline is on Neutron. It is our entry into OpenStack. One problem is L2 underlay, because it doesn’t scale. Second problem is Overlay. Upstreaming code is a challenge.

Lacour: We are trying to build public cloud at Dreamhost. Dreamhost has always been built on open source. L2 virtualization is provided by VMware. L3 and above is implemented internally. Most of the 300,000 customers are end users, not enterprises. Today Neutron isn’t up to speed with IPv6 though Juno is getting there. To have intelligent overlays and cheap fast dumb underlays is great for us.

Balland: Working on high level abstractions. Congress. VMware has had full-time developers dedicated to Neutron and Quantum before that. Geneve will replace GRE.

Next, under Emerging Trends, was NFV in OpenStack, featuring

Martin Taylor| Metaswitch \ Calico
Alan Kavanagh | Ericsson
Kyle MacDonald | OpenNile
Jennifer Lin | Juniper
Chris Wright | Red Hat

MacDonald: ETIS is European Telco Standards Institute, wrote a manifesto for all of their vendors at 2018 to deliver their network gear as Software instead of aggregated systems. Started in 2012. When telcos have the agility of virtualization, they can achieve a lot. Lot of POCs so far.

Taylor: VNF performs the means of a telco in software. Traditionally telcos demanded standards and would wait 5 yrs or so. Now they no longer have the luxury to wait and need to get stuff done now. ETSI has defined a framework wherere vendors and Network Operators can get together and demonstrate some capability to drive the industry forward. We already have a virtualized session board controller in Europe. Challenge of network people is to convince carrier that the solution is carrier grade. Carriers feel that OpenStack has some challenges, such as packet processing. Telcos feel that their ability to innovate is hindered by the very small number of vendors they work with. In the ETSI manifesto, they say that they can source workloads from a much broader ecosystems, who are more nimble. Telcos have learned from the lesson of WhatsApp who facilitate free text messages and then got bought for $19B as an example of a competitor. Telcos feel they need to get up to speed.

Wright: Service chaining is the ability to steer packets through a collection of services such as a set of logical services (Firewalls, Load Balancers, IDS, etc) in the proper order. SDN steers the packets, Service chaining is how an SP will package these services to customers. Service Providers want a general purpose compute platform, and that is a cloud problem. ODL has OpenStack integration that can help NFV. There is also a service chaining project. Overall ETSI NFV project identifies a suite of open source technologies such as from DPDK and ODL. To have elastic flexible infrastructure is the biggest gain to shift from CapEx to OpEx.

Kavanagh: Telco vendors must be able to pick and choose various servers on vendor boxes and chain them together ala service chaining. Ericsson is virtualizing their entire product portfolio. Challenge is the time to market. NFV will shrink lead times tremendously. ODL is a good complement to Neutron; we will see more advanced services like BGP being added. Telcos are looking into NFV to reduce the lead time. and virtualize their key services to compete with Amazon. They also see OpenStack as a good fit because it also reduces costs.

Lin: Diff b/w SDN and NFV: SDN is about automating and orchestrating SW-driven platform. ETSI has unique applications, like stateful apps. how to tie into OSSP systems, how to handle latency-sensitive apps that are cannot exist in a stateless form. We should not reinvent the wheel, e.g. IP VPN. Contrail technology is based on a standard that was co-authored by AT&T and Verizon. NFV is more service orchestration and service delivery than how do we transport packets. Challenge is how do we expOpenStacke this as a service and improve the orchestration.

The final panel in the afternoon track I attended was Four SDN Use Cases, with

Lane Patterson | Symantec
Jonathan LaCour | DreamhOpenStackt
Adam Johnson | Midokura
Ben Lin | VMWare

Lin: VMware is running its Internal cloud on OpenStack since 2012, which is used for development, test, demos, pretty much everything. Ebay is one of their first customers. NSX is multi-hypervisor, hence OpenStack. Main pane of glass is vSphere. With so many moving parts in OpenStack, it is important to understand the amount of scale you’re looking for. Out of the box, they get great performance with STT, investigating DPDK. They can send traffic directly from hypervisor to hypervisor, which reduces traffic a lot.

LaCour: He spoke of a Dream Host OpenStack-based cloud offering called Dream Compute, mass market public cloud offering that has IPv6 Dual Stack support for customers as well as back-end systems, L2 isolations for every tenant, and drives cost down (went with white box switches running Cumulus Linux so that their operating team could be generalist. Spine and leaf 40G switches b/w spine and leaf. 10G access switches, 10G storage networks (i.e. storage based on ). They believe that overlays enable simpler physical networks. Their overlays are with VMware NSX, and they were one of the first Nicira customers. They used them because it had the most solid L2 support. They don’t use their L3 solutions because they didn’t have supoprt at the time, so they implemented their own, using Akanda (a router virtual appliance that has Neutron extensions). Their L3 service appliance uses iptables in Linux. Storage networks are separate from data networks, minimal overhead. Each tenant gets a single router appliance that is booted independently and ephemerally and can be as small as 256MB, 512MB with more features. They expect Dream Compute to go GA next year.

Johnson: Midokura does Network Virtualization Overlays for OpenStack. KVH is the customer, member of the Fidelity group, providing Managed Private Cloud for financial services customers. MidoNet is an OpenStack overlay for isolated private cloud environments that provides distributed features (Load Balancing, Firewall, routing), not just distributed L2 services. It programs the Linux kernel to do packet processing; uses VXLAN. They saturate their Mellanox 40G uplinks with 37G.

Patterson: Symantec grew by acquisition with several BUs and 58 DCs around the world. They consolidated many of them via OpenStack. They offer services such as object storage, batch processing, authentication, etc, i.e. more than just IaaS. They use Bare metal for their Hadoop and Cassandra + OpenStack Nova environment. For the SDN layer, they used something to supplement Neutron, and ended up going with Contrail for its MPLS over GRE capability, though they are moving to VxLAN. To speak to the bare metal side, they used a Docker-like layer so that each tenant on the Computer virtualization side has to authenticate before talking to the Big Data side. In one environment, for leaf-spine, they use Cisco and in another they use Arista with only generic vendor agnostic features being implemented. TORs are 4x40G into the spine, spine is a 2x, moving into a 4x. When they used generic OVS, every vendor outperformed OVS in the bakeoffs. They saturated the uplinks from the TORs to the spines.

Interestingly, none of the 4 panelists used OpenFlow at all in their clouds.

OpenStack

Live Blog of OpenStack Silicon Valley 2014 Community Event live blog Part 1

September 15, 2014 Umair Hoodbhoy 2 Comments

This is a live blog of the inaugural OpenStack Silicon Valley Community Event, held at the Computer History Museum in Mountain View, California on September 16, 2014. It was put together by Mirantis, one of the biggest proponents of OpenStack.

The Agenda and Megathemes for the Day is given by Alex Freedland, Mirantis Co-founder and Board Member of OpenStack Foundation.

Freedland states that as someone who’s been involved with OpenStack since the beginning, four years ago, he has noticed a distinct qualitative change in OpenStack. We no longer have to prove that OpenFlow will win the open cloud game. It is here to stay and will be the open cloud platform of the future. The question we have to ask is “What will it look like when it matures and how will it appear to enterprises, service providers, etc?” Freedland feels we are entering OpenStack 2.0 and this event is about OS 2.0. Now that we are in the 2.0 territory, we are seeing a lot of huge deals, with some large companies entering at massive scale. The usage patterns we see are of agility and management. It’s necessary for Software Defined Economy to detect change quicker. In just four years, OpenStack is at least as large as Linux.

Next, up is a keynote by Martin Fink, EVP and CTO, HP and a late inclusion, Marten Mickos, CEO, Eucalyptus. The title was Looking ahead: OpenStack, Eucalyptus, Open Source, Enterprise and the cloud?

As HP recently announced plans to acquire Eucalyptus. Martin Fink and Marten Mickos present a look ahead at what this holds for open source, the cloud, the enterprise and OpenStack. Eucalyptus is open source private cloud software that supports industry standard AWS APIs and creates cloud resources for compute, network, and storage.

Martin Fink: “Beginning Nov/Dec last year, there was a massive swing of HP contributions towards OpenStack. Helion launched in May 2014. Today with Juno, HP is the No. #1 contributor to OpenStack. When we started working on open cloud, we felt we had to do it in the true spirit of open source, getting very close to trunk. When I was running the open source division at HP between 1999 and 2005, the funnest part was being in a room with competitors, like IBM, and working towards the overall benefit of open community. I made some friends along the way. We were not confused about when it was time to compete. HP and Mirantis compete in some areas, but today we are together. We’re not just delivering a distribution or a piece of software. We’re delivering the actual distro, the hardware along with it, and we deliver it any way you want: private, VPC, hybrid, public, you name it.”

Marten Mickos then comes on. The acquisition has not yet been finalized: Open source will win in Cloud by being modular, having competitors collaborate, having anyone to scrutinize & improve quality, by having Darwinism (good stuff survives), and by having less lock-in. Open source will have challenges too: for example, there has to be someone who says NO and members compete with each other.” Also, “OpenStack and Eucalyptus is where Nimble (Eucalyptus) meets massive (OpenStack). Eucalyptus is a tiny group. It is also Hybrid with AWS design patterns in the open. In public cloud, AWS API is private. In private cloud, AWS API is public. It is critical in cloud for the core pieces to remain hardened. OpenStack can and will have components, add-ons, adjuncts and alternative projects (Ceph, RiakCS, Midonet) and the aim of Eucalyptus is to become one.”

Next is a lightning talk by Ken Ross | Director of Product Management, Brocade

Ross says Brocade has been involved with OpenStack for 3 years and has increased its level of investment. Brocade’s contributions include SAN FibreChannel for Cinder, Multitenant DC-DC via MPLS, and NFV ETSI POCs inc. scheduling, FWaaS, and VPNaaS. 80% of NFV discussions are centered around OpenStack. Challenges: Neutron maturity – it has evolving functionality in cusotmer engagements (big spectrum of customer asks across releases, Folsom, Grizzly, Havana, and Icehouse, which makes it extremely difficult to be agile in development). Another challenge is for the SDN community and Open Daylight community to understand each other better.

This is followed by a keynote on The Software Defined Economy by Jonathan Bryce Executive Director, OpenStack Foundation

No matter what size your organization is, it’s not moving fast enough. Software innovation is make-or-break. If infrastructure isn’t part of the solution, it’s part of the problem.

Bryce says “Every company is competing with a startup. E.g. in banking you have to go against Stripe, Swuare, PayPal, etc. In Big Media, you have Netflix (won an Emmy), Zynga. In Automotive, Tesla, Uber, Lyft, SpaceX (which forces the US Airforce to compete with a startup). This is the Software Defined Economy. It is the ability to change easily, quickly from one vendor to another. The Old model was passive consumption – we bought what our vendors sold us and upgraded when they told us to upgrade; it was ok to use multi-year product cycles). The New Model is: I want what I want now (for example, mix and match in a single DC, release early and release often, deploy directly to production. Be agile, BYOD). Technology decisions are moving out to the edges of the business. Cloud is being driven from the edges of the business. It removes barriers and allows innovation. Quote from Disney: “In an IT department, you have to think like a Product company.” Top 10 car company used OpenStack to harness Big Data from dealer reports, insurance filings, car sensors, and generate reports to supply to various departments (R&D, Sales, Marketing)

Time for the keynote from Martin Casado | CTO of Networking, VMware on Policy for the Cloud Frontier. He comes to the stage with extreme energy, as if he just sprinted 100 meters.

Casado: Automation is great. Everyone loves the promise of all cloud operations codified as running programs tirelessly doing the work of dozens of operators. Unfortunately, today we can only automate infrastructure and applications——ideals for cloud behavior that are more concerned with business logic, regulations, security, risk management, and cost optimization than infrastructure and applications. As a result, the policy layer presents a new challenge in our quest for cloud automation. In this talk, I will discuss the policy problem and why it is an emerging area we should focus on. I will then discuss Congress, an OpenStack effort to create an open policy framework to help us as a community step into this new frontier.

Casado: “Automation does not remove the human. Automation is necessary but insufficient for removing humans from the cloud control loop. Humans still need to interact with cloud to make it obey business policies. Policy is the Holy Grail of IT. The Policy problem is as follows: Humans have ideas, humans can document ideas, systems don’t understand human languages. Non-technical people with non-technical ideas want to dictate how the system works. Somehow we’re supposed to get this to work on the backend. How OpenStack can crack the problem is as follow: Computer Scientists will want to write a declarative language, which will need a compiler to implement this in the System. But such policy systems have always existed and have their flaws. Traditional barriers are 1. Device Canonicalization (lowest common denominators are Cisco, Juniper, Brocade, Arista, etc) that fail because of the interoperability, 2. Distributed State Management (e.g. at 5 am person XYZ is probably not in a proper state of mind, so don’t give him access), and 3. Topology Independence (if you choose a language that is independent of toplogy, you require mapping from physical topology to logical, which is very difficult). Many of these problems have been solved by OpenStack because of abstractions (no more device canonicalization problem). Software abstraction has been canonicalized. Primary value of A CMS is to present a consistent view, manipulate. Policy compiler can now exist over this with this level of abstraction present.

He then defines Congress: An open policy framework for automated IT infrastructure. Every component (networking, storage, security, compute) has a policy layer. But they can only be used for that one component. High level framework has to unify them. it is an enormous hurdle to adoption to translate them. E.g. Application Developer, Cloud Operator, and Compliance Officer have separate ideas, but should be limited by the laws of 1 language. That’s what Congress aims to achieve. This is how it is the Holy Grail.

Randy Bias | CEO and Founder, Cloudscaling then spoke on The lie of the Benevolent Dictator; the truth of a working democratic meritocracy

Bias spoke of the groups with longer term vision (strategic business direction) and tactical teams with shorter term focus (release dev lifecycle). OpenStack doesn’t have a vision or product strategy. There are so many new projects coming up that it changes the meaning of OpenStack. But there is no shared vision or ownership at the top. By product strategy for open source projects, we need product leadership, not a benevolent dictator. Some requirements include: managing it like a product (even though it isn’t one); focus on end-user needs and requirements; have long term vision and long term prioritization & planning; have corporate independence; closer workings b/w Board and TC; and architectural oversight and leadership. He respects AWS in the sense that it has a small architectural review board, a product management function (team of PMs per product line). Suggests having something similar: (Architecture Review Board elected for 2-4 years, having a wide set of domain expertise; and PM function with specific domain expertise.

This was followed by a Lightning talk by Chris Kemp | Founder and Chief Strategy Officer, Nebula

Kemp said that the interoperability b/w products and services is what will make OpenStack successful in the long run. The goal should be to have zero consultants, zero additional headcount. Movie studios using biotech companies, space agencies, using Nebula solution.

The final keynote was by Adrian Ionel | CEO, Mirantis on OpenStack 2016: Boom or Bust?

Ionel spoke of the growth adoption of OpenStack. He said that the measure of success is actual workloads. AWS is $6B business. Collectively OpenStack doesn’t even scratch the surface of that number. Docker has had 20M downloads over the past 4 months. In the end, Developers win. They are the vanguards. They don’t care about deployment choice of monitoring software, which hypervisor is used underneath, what the scalability of the network layer is, or which storage system is used for volumes. What they do care about is API quality and ease of use, feature velocity, and portability. He suggests focusing on the APIs as the key to adoption, investing in ease of use vs even more flexible plumbing, not moving up the stack (e.g. XaaS) but rather partnering instead, reshaping upstream engineering (i.e. technical committees) to foster open competition (vs central plumbing), and enabling workload mobility to other platforms. Mirantis signs 2 new customers a week, however, it is early days yet. And OpenStack needs to be able to scale appropriately.

Three fireside chats followed. The first was Is open source cloud a winner-take-all game?

Gary Chen (Moderator) | Research Manager, IDC
Marten Mickos | CEO, Eucalytpus
Steve Wilson | VP & Product Unit Manager Cloud Platforms, Citrix Cloudstack
Boris Renski | Mirantis

Mickos: In a digital world with exponential development, you do see Winner-Take-All examples, e.g. Linux and mySQL. But exceptions exist. He quoted Linux Torvalds as “If Linux kills Microsoft, it will be an unintended consequence”. Customers just want value. Not many companies have the depth and breadth of skill that HP has. They believe in hybrid clouds; that’s why it stands out.

Wilson: Innovation isn’t a zero-sum game. There are different solutions for different people. If you declare victory at this point, you’re living in a bubble. You’re ignoring AWS, VMware. It is heterogeneous by definition. CloudStack is the most active solution in the Apache community. It doesn’t make sense to have a winner-take-all game. I don’t think OpenStack and CloudStack compete with each other. CloudStack is very easy to use and works at scale. Thousands of CloudStack deployments exist around the world. The Netscaler and Citrix team already contributes to the OpenStack world. Xen contributes to hypervisor component of OpenStack. We will work with whatever cloud infrastructure our customers use.

Renski: There is an opportunity for only 1 standard open source cloud. There is only space for 1 winner. Enterprises and technology-centric organizations invest a lot in their infrastructure and OpenStack is the only way to solve their problems. The sheer number of vendors who have come together to solve problems with minimal disruptions to the tenants is the core advantage of OpenStack.

The next Fireside chat, titled, If OpenStack is so awesome, why doesn’t every enterprise run on it?, featured:

Chris Kemp | Founder and Chief Strategy Officer, Nebula
Peter ffoulkes | Research Director, Servers and Virtualization and Cloud Computing
Jo Maitland | Analyst for Google Cloud Platform
Alessandro Perilli | VP & GM, Open Hybrid Cloud, Red Hat

Ffoulkes Enterprises are moving slowly and don’t like lock-in. OpenStack is moving fast. There will be mergers and acquisitions. Customers’ have difference challenges, e.g in one country deployment can be on-premises, in another must be off-premises. There’s a high cost and complex for enterprises to build DCs of their own or have private clouds. It is going to be a slow journey for customers in non-IT verticals to migrate to public clouds.

Maitland OpenStack hasn’t lived up to process of workload portability, but containers will help as they are super efficient and fast. The response to Docker is encouraging. As soon as there is a single semantic deployment model for OpenStack, the floodgates will open on the enterprise on-premises side. But it is a gradual migration.

Perilli There is a massive gap b/w expectations and reality. Customers ask if OpenStack is a cheap version of a virtualization layer that they can get from other vendors? That is a misperception. Vendors are not preaching OpenStack properly; they are confusing and scaring customers by saying this is the new world. They need to transition from scale-up model to scale out model. You need to have something on top that glues the scale-out with the scale-up. Enterprises generally take a long time to make such decisions. In order to increase adoption, what’s missing from OpenStack is the capability to think beyond what OpenStack can offer. It needs to be coupled with other layers that can be merged with OpenStack in a seamless way to enforce policies that large enterprises need. We’re still looking at the foundation.

The Final Fireside chat of the morning session, titled Open Source Business Models: Open Core, Open Everything or…, featured:

Jonathan Bryce (Moderator) | Executive Director, OpenStack Foundation
Jonathan Donaldson | General Manager, Software Defined Infrastructure, Intel
Brian Gentile | VP & GM, Tibco Jaspersoft
Nati Shalom | CTO & Founder, Gigaspaces

Shalom: There are different reasons why open source projects succeed and fail. e.g. mySQL succeeded because it entered a proprietary space with a cheap solution that delivered Alternatives were expensive. Also, Docker went from disruption to commodity in 6 months or less. They entered a very small space that few people were addressing, and was timely. For OpenStack, RackSpace realized that they couldn’t compete very long with AWS. They built a coalition with OpenStack. You go into coalitions because there is no other option. Open Source changes the dynamics in that things that are commodities should be free, and those that add value should be paid for. With Android, Google created the gravity that allows allows open source developers to contribute. It is not healthy for companies to argue for equal rights.

Gentile: Open Source needs to attain an unmet need. The development of distribution methodologies of open source is superior than proprietary ways. If all the physicists of the world held tight to their discoveries and work, where would we be?

Donaldson: Mass adoption from a community perspective must comes with acceleration. Enterprise customers often like licensed model, which might work. Somebody has to pay the bills though.

This concluded the morning session. I will cover the afternoon session in Part 2.

Umair Hoodbhoy – Multi-Cloud, Networking, and Kubernetes

Monthly Archives: September 2014