Category: Virtualization

Mrs. Y, the network security princess, sent me an interesting design challenge:

We’re building a private cloud and I'm pushing for keeping east/west traffic inside the cloud. What are your opinions on the pros/cons of keeping east/west traffic in the cloud vs. letting it exit for security/routing?

Short answer: it depends.

Dumlu Timuralp (@dumlutimuralp) sent me an excellent question:

I always get confused when thinking about IP multicast traffic over VXLAN tunnels. Since VXLAN already uses a Multicast Group for layer-2 flooding, I guess all VTEPs would have to receive the multicast traffic from a VM, as it appears as L2 multicast. Am I missing something?

Short answer: no, you’re absolutely right. IP multicast over VXLAN is clearly suboptimal.

Arista is launching a new product line today shrouded in mists of SDN and cloud buzzwords: the 7150 series top-of-rack switches. As expected, the switches offer up to 64 10GE ports with wire speed L2 and L3 forwarding and 400 nanosecond(!) latency.

Also expected from Arista: unexpected creativity. Instead of providing a 40GE port on the switch that can be split into four 10GE ports with a breakout cable (like everyone else is doing), these switches group four physical 10GE SFP+ ports into a native 40GE (not 4x10GE LAG) interface.

But wait, there’s more...

A while ago I described the need for BPDU guard in hypervisor switches, and not surprisingly got a number of “it’s there” tweets seconds after vSphere 5.1 (which includes BPDU filter) was launched. Rickard Nobel also did a magnificent job of replicating the problem my blog post is describing and verifying vSphere 5.1 stops a BPDU denial-of-service attack.

Unfortunately, BPDU filter is not the same feature as BPDU guard. Here’s why.

Almost everyone agrees the current way of implementing virtual networks with dumb hypervisor switches and top-of-rack kludges (including Edge Virtual Bridging – EVB or 802.1Qbg – and 802.1BR) doesn’t scale. Most people working in the field (with the notable exception of some hardware vendors busy protecting their turfs in the NVO3 IETF working group) also agree virtual networks running as applications on top of IP fabric are the only reasonable way to go ... but that’s all they currently agree upon.

When VXLAN came out a year ago, a lot of us looked at the packet format and wondered why Cisco and VMware decided to use UDP instead of more commonly used GRE. One explanation was evident: UDP port numbers give you more entropy that you can use in 5-tuple-based load balancing. The other explanation looked even more promising: VXLAN and OTV use very similar packet format, so the hardware already doing OTV encapsulation (Nexus 7000) could be used to do VXLAN termination. Boy have we been suckered.

Update 2015-07-12: NX-OS 7.2.0 supports OTV encapsulation with VXLAN-like headers on F3 linecards. See OTV UDP Encapsulation for more details (HT: Nik Geyer).

A while ago, a tweet praising the wonders of 802.1BR piqued my curiosity. I couldn’t resist downloading the latest draft and spending a few hours trying to decipher IEEE language (as far as the IEEE drafts go, 802.1BR is highly readable) ... and it was déjà vu all over again.

Short summary: 802.1BR is repackaged and enhanced 802.1Qbh (or the standardized version of VM-FEX). There’s nothing fundamentally new that would have excited me.

A few years before MPLS/VPN was invented, I’d worked with a service provider who wanted to offer L3-based (peer-to-peer) VPN service to their clients. Having a single forwarding table in the PE-routers, they had to be very creative and used ACLs to provide customer isolation (you’ll find more details in the Shared-router Approach to Peer-to-peer VPN Model section of my MPLS/VPN Architectures book).

Now, what does that have to do with OpenFlow, SDN, Floodlight and Quantum?

Straight from RFC 6670 (section 3.4):

[...] as is usually the case with communications technologies, simplification in one element of the system introduces an increase (possibly a non-linear one) in complexity elsewhere. This creates the "squashed sausage" effect, where reduction in complexity at one place leads to significant increase in complexity at a remote location.

This is probably the most concise description of the great idea of using long-distance vMotion for “mission-critical” craplications, and applies equally well to the kludges used to compensate the simplicity of virtual switches.

Almost a year ago, I predicted that eventually the hypervisor vendors will wake up and realize it’s time to get rid of VLANs and decouple virtual networks from the physical world. We’ve got the first glimpse of the brave new world a few weeks after that post was published with the VXLAN launch, but that was still a Cisco’s solution running on top of VMware’s (and now everyone else’s) hypervisor. The recent VMware’s acquisition of Nicira proves that VMware finally woke up big time.