On MPLS Paths, Tunnels and Interfaces
One of my readers attempted to implement a multi-vendor multicast VPN over MPLS but failed. As a good network engineer, he tried various duct tapes but found that the only working one was a GRE tunnel within a VRF, resulting in considerable frustration. In his own words:
How is a GRE tunnel different compared to an MPLS LSP? I feel like conceptually, they kind of do the same thing. They just tunnel traffic by wrapping it with another header (one being IP/GRE, the other being MPLS).
Instead of going down the “how many angels are dancing on this pin” rabbit hole (also known as “Is MPLS tunneling?”), let’s focus on the fundamental differences between GRE/IPsec/VXLAN tunnels and MPLS paths.
- Tunnels (IP-over-IP, IPv6-over-IPv4, GRE, IPsec, VXLAN…) were designed to connect discontiguous bits of an overlay network across an underlay network without the underlay network ever seeing the packets
- MPLS paths were designed to be a better packet forwarding mechanism inside the same network.
Because we’re using tunnels to connect segments of a larger network, we must exchange reachability information between those segments. Cisco once tried to do that without the tunnel interfaces (crypto maps anyone?), but fortunately quickly came to their senses (VTI interfaces). Running a routing protocol over tunnel interfaces (while trying to avoid the recursive routing blowbacks) remains the best way to build an overlay network.
You were not supposed to run routing protocols over MPLS paths (don’t get me started on the MPLS TE tunnel adjacency nightmare). The network was supposed to figure out where things are using regular routing protocols, use the results of those calculations to set up MPLS paths (LDP), and then map the forwarding equivalence classes (prefixes) onto MPLS paths.
Every now and then, I bump into someone claiming that Tag Switching1 was about traffic engineering or VPNs. That’s simply not true; the initial use case was improved forwarding performance (in particular, using ATM switches in IP network).
Straight from RFC 2105:
Tag switching blends the flexibility and rich functionality provided by Network Layer routing with the simplicity provided by the label swapping forwarding paradigm. The simplicity of the tag switching forwarding paradigm (label swapping) enables improved forwarding performance, while maintaining competitive price/performance.
-
Cisco’s technology that went through the racing-horse-to-camel transformation within the IETF and became MPLS. ↩︎
The MPLS paths used to implement IP forwarding were always created on demand based on the contents of the IP routing table. Creating dynamic interfaces to map MPLS paths into would be an interesting exercise in futility. The ability to merge multiple MPLS paths to the same egress device anywhere in the network1 would make the implementation of that idea even more interesting.
On the other hand, I was a bit surprised when the MPLS TE tunnel interfaces turned out to be unidirectional constructs2. Preventing recursive routing and providing network designers with as much flexibility as possible3 likely outweighed any potential drawbacks.
Finally, you’ll find even more details in the MPLS is not tunneling blog post from 2011.
-
If you like digging into the gory details, you’ll love the Label Merging section of the Multiprotocol Label Switching Architecture RFC (RFC 3031). ↩︎
-
Apart from the RSVP reservations, the tail-end LSR has no idea it’s the egress point of an MPLS TE tunnel. ↩︎
-
The traffic engineering tunnels are unidirectional, allowing you to have different paths in each direction. That minor detail derailed ITU to the point where they wanted to build their own version of MPLS and then settled on MPLS-TP. ↩︎
And I still don't agree.
The idea for tag-switching started supposedly in 1995. I joined cisco engineering in the summer of 1996. The first tag-switching BOF at the IETF was in December 1996. https://www.ietf.org/proceedings/37/ I was in Europe at the time (cisco's first over-sea's DE). So I am sure I wasn't as up-to-date as others.
But this is what I heard:
Large ISPs in the US were using an ATM underlay. Because they could do some form of TE in their ATM underlay. They paid Fore Systems a lot of money. Fore was the preferred ATM vendor at the time. Some people at cisco thought that money should go to cisco.
This is what I saw: In 1996, Ipsilon was making a lot of noise with their "IP Switching". MPLS killed Ipsilon.
It turned out that the real killed-app for MPLS was BGP-MPLS-VPNs. That technology allowed the ISPs to sell a new service. And make lots of money off that service. I read somewhere (around 2012-2015) that 0.5% of ISPs worldwide were using MPLS-RSVP-TE. And 0.0% of enterprises worldwide. So imho, BGP-MPLS-VPNs is the only MPLS-related technology that really mattered.
Of course everyone's first idea was: "when you do index-based forwarding, that must be faster than a short-length prefix lookup". It turned out it wasn't. People knew this very early on. Before any MPLS product shipped. I knew it, and I've never been involved with forwarding code myself. I'm sure you still don't believe me. So have a look at what Bruce Davie wrote:
https://systemsapproach.org/2022/02/14/was-mpls-necessary/
Hey, I expected you to disagree, but even the slides from the Tag Switching BOF talk, somewhere deep inside those proceedings you shared, focus primarily on performance and scalability, and the RFC 2105 was published in 1997, so one would hope they would have a clear picture by then.
Anyway, Tag Switching appeared in the pre-7500 days when the best Cisco could offer was Cisco 7000 with the CBUS complex (peaking @ 500 Mbps according to https://newsroom.cisco.com/c/r/newsroom/en/us/a/y1995/m08/cisco-7500-series-boosts-routing-muscle-for-large-evolving-corporate-networks.html). Low-end ATM switches (Fore and Cisco's Lightstream 1010, which was indeed really late to the market) got to 5+ Gbps for a significantly lower price. Cisco 7500 (launched in 1995) got to 2.1 Gbps (same source), which was still 2.5 times slower than an entry-level ATM switch. StrataCOM switches were a totally different beast.
I also won't argue that by the time someone finally decided to build a packet-forwarding label-switching linecard, the hardware was fast enough to make little difference between IP longest-prefix match and label lookup.
I agree TE was one of the early use cases, but it was a static hop-by-hop list of IP addresses. Constraint-based routing with IS-IS and OSPF came way later (a release or two after MPLS/VPN). The whole thing looked so ridiculous that I couldn't understand why anyone would be using it, until I was told some big ISPs periodically calculated explicit paths in a network management software and downloaded them to router configs.
I also totally agree that for most people, the first useful use case was MPLS/VPN until we got to the "OMG, full BGP table has hundreds of thousands of prefixes" moment, at which point the BGP-free core started to look interesting.
Label switching is routed in the old telephony network. You had the DCN and a network management node that calculated the various network paths. Then it loaded the routing tables into the telephony switches. The trunks had local labels that were used in those routing tables. This is the root of label switching solutions.
Then we had the Synchronous Transfer Network (STM). Sonet/SDH and PDH in a hierarchy. ATM was just introducing a data plane with statistical multiplexing. But the logic was the same for some time as it was in telephony switches.
In 1996 I had to build a network for the JENC conference from ATM equipment provided by 7 (!) different vendors. All my university colleagues escaped this challenge, so I had to build this metropolitan area network over Budapest with my 5th grade students. From those 7 vendors only one supported SVCs. All the others had only PVCs with static routing tables that had to be entered manually.
When these telco vendors were forced to come up with some IP based solutions, they were quite happy to use label switching, since it was so similar to their previous voice and STM/ATM architectures. They just had to add a different control plane.
Remember the Cisco acquired Stratacom ATM switches having a separate node as a PNNI controller for SVCs. The basic switch was still PVC based ATM with simple label switching.
For the switch vendors label switching made it possible to create very simple core switches. BGP free was the selling term. The core switches could focus on forwarding performance using hardware acceleration and you could ignore the complexities of the usual MPLS/VPN IP control plane.
But as the hardware improved, the original advantage of fast label lookups versus slow full IP address lookups and smaller memory needs faded away.
After some years my colleagues even split the telco network into two parallel networks: a multi-service segment that is complicated, MPLS based, expensive, limited capacity and another separate cheap and simple, high-capacity flat IP network for public internet services without labels, MPLS and VPNs. This architecture practically killed the business case for the DT preferred Terrastream. It was so much simpler and cheaper...
BTW, SDN with Openflow was just a recast of the old telephony network. PCE is another such revival...
Having read the MPLS is not tunneling article it makes perfect sense to me, but then, what the heck is SRv6 lol. Seems like a mix of both if I understand it right
> what the heck is SRv6?
Another attempt to develop a unifying standard (see https://xkcd.com/927/)? On a more serious note, if you use it like you'd use MPLS paths or as the second label in L3VPN, it's pretty similar to MPLS and doesn't create interfaces on (some) network devices. Obviously, you could also use it to create tunnels...