Join us as we gather around the hedge,
where we dig into technology, business, and culture
with the finest minds in computer networking.
Well, good morning,
Audrey.
Audrey says hi.
Hi. Hi, Tom.
Oh, hi. Oh, yeah. I'm here too. I'm
glad to see that you joined Audrey this
morning for this episode of The Hedge.
Audrey's the plant.
Got it. I thought I remembered the plant
reference. Yeah. Yeah. I have I have virtual
plants, so I came prepared. I don't have
I don't have names, though. So sorry.
That's okay.
And this morning, we're joined by Scott Robyn.
By the way, Tom, are you warm?
Somewhat. I mean, it's gonna be, like, 65
today. So
Oh. Oh, well, no. I don't know. Just
trying to make everybody jealous who's there. It
works. You just did. You just did.
Scott, where are you? I don't remember where
you are. I am in Harrisonburg,
Virginia in the beautiful Shenandoah Valley, not that
far from you. So Yeah. Not that far.
I think it's still a bit colder and
snowier there than it is for me. Maybe.
I don't know. We're we're we're going below
zero at night here still. So Yeah. Well,
we are too. I mean, we do it.
Yeah. We still do in in,
in Knoxville. It's still pretty darn cold in
Knoxville, and they aren't planning on some snow
this weekend because they brined the roads.
Like, whatever. Nobody's gonna drive anyway.
Like, we're all staying home. Although, I may
go try to go to the gym on
Sunday even though it snows. We'll see if
the gym is open.
That'll be the challenge. Will the gym open?
I I doubt it. So today, we are
talking about MPLS, not the city,
the protocol.
And there are many,
many ways to say this, mostly pointless label
switching. What did you say, Tom? Mipples?
Mipples?
It's a little cringey, but, you know,
I can tell you're entertained, Russ. I never
had a problem with MPLS. Like, it's not
too many syllables.
Right? I never felt the need to shorten
that. Four syllables.
Yeah.
I mean,
you know, it's kind of counterintuitive
because the entire point of MPLS
is to have a shortened
forwarding
paradigm.
Do not have to forward on the entire
IP v six address or IP v four
address, but there's, like, this little short little
label. So, you know
and somehow we end up with IP, which
is only two syllables,
and it's shorter to switch based on MPLS,
which is well, I I don't know. That's
an unfortunate anti coincidence or whatever one would
call that, but,
yeah.
Very strange. That's a great that's a great
place to dive in. Like, you know, okay.
What's the problem that was trying to solve?
Right? And,
you know,
look so one one person's perspective,
you know, with incredible
growth of the Internet
in the mid nineties into the late nineties,
we had lots of issues with,
you know, centralized forwarding and software based forwarding.
And just sheer forwarding tables, by the way,
the size of a forwarding forwarding plane. That
size size of forwarding table
and the ability to, you know, have multiple
d s threes
in a single router, you know, or multiple
o c threes.
You know, yes, we're dating ourselves
here. But,
that computational complexity involved with best match what's
that?
Frame relay.
Yeah. Sure. Yeah. Yeah. It's switching. It is
switching. Yeah.
We can go there if you want.
But but the that idea
of forwarding based on fixed length labels that
are always a unique match
versus longest match routing and IP, which has
great
benefits,
you know. However, you know, it's more computationally
complex. You know, the the thought arise that,
you know, if we just took an old
switching paradigm,
and bit forwarded packets based on those fixed
length labels, we can get better throughput in
boxes and keep up with the growth of
the Internet. We also have to remember that
at this time, when MPLS came out, when
Yankov started working on tag switching as we
called it Gotta start there. Yep. Yeah.
ATM, you go back to the Gartner charts,
and ATM was up into the right. We
were gonna have 25 gig
AT or was it Meg?
Maybe it was Meg at the time. It
was Meg at the time.
ATM to the desktop,
and it was going to solve our voice
over IP problems Sure. Because voice can't run
over IP.
It must be on a circuit
based system,
not on a sarcastic packet based system. Like,
you just you can't do this stuff. Right.
And so, you know, MPLS also came a
little bit or a lot out of that
kind of
environmental thinking.
Well, full disclosure,
I I was a four systems trainer, an
ATM switch manufacturer,
before I was a Cisco trainer.
So, I I was talking about NSAP addresses
and RFC fifteen seventy seven,
before I really even learned BGP.
So
Yeah. Okay. That explains
a lot of that.
That's good. So
do you remember the Cisco 8,500?
I The the old 8500?
So does that have a connection to the
Lightstream acquisition?
Because I knew the Lightstream ten ten
and the the ten ten integration into the
Cat 5,500.
Yes. And and I deployed a bunch
of 55 hundreds
at Bell Atlantic way back. So take so
take an take a Lightstream ATM switch. Mhmm.
Put an RP
in the box from a Cisco 7,500
and call it a router.
Sure.
Yep.
Like, man, did that box have a lot
of limitations?
We actually had to invent
games for the MAC address because it didn't
have a large enough MAC address table to
support
in host or larger devices being attached to
the ports.
Like,
we had this whole thing called the adopt
a Mac hack
that we've patented to get to, like, get
around those problems. But, anyway okay.
So that's the origin of a t of
MPLS. I almost said ATM there. Well, anyway
Well, there's there's another another fun,
you know, bash there. You know, what is
MPLS?
It's bringing the complexity of ATM circuit switching
to IP. Yeah. So Yeah. Exactly. Yeah. And
so so those are all the things. Now
if we were to talk about the good
side of MPLS though, first of all, is
MPLS still widely used?
Yes.
Things don't get turned off very quickly in
in most networks, especially here at That was
a kind way to put it, Scott. Thank
you.
Yeah. No. It really is. A lot of
people still use MPLS, but largely for late
for for,
traffic engineering, traffic management, traffic steering
over transit networks, I would say, is the
primary place that I see
MPLS engaged today.
And it's still used for l two VPN,
l three VPN services
no matter
how many times
salesmen say MPLS is dead.
No.
Yeah. MPLS based services are not dead. They're
still out there, and they're still very common.
Well, I think there's an an interesting point
to pick out there on on both of
those applications.
So we started out
trying to keep up with the growth the
bandwidth growth of the Internet core, basically.
And you can correct me if I'm wrong,
but I don't think MPLS VPNs
or traffic engineering
were necessarily those initial goals. And there are
application areas that emerge after,
you know, hey, we found I've got this
label that can be a shim and let
me replicate address space in multiple,
you know, VPNs.
So that was was another one area that
emerged. And then the ability
to very granularly,
like, through through my network,
get better utilization out of underutilized links, you
know, do it very strictly, do it very
loosely.
That's that
that is a secondary
area that also emerged. That's that's my perspective.
Yeah. Yeah. It did not start out. It
started out attacking
the same problem,
that Dino was trying to attack with Lisp,
which which was just sheer the amount of
memory required
on a forwarding device
and the amount of complexity
in building,
a chip that could forward based on
variable length.
That's that's really I mean, and honestly, people
say, why did you have classful? You know,
why did we have classful forwarding in the
early Internet? It's exactly the same answer.
In the early Internet, everything was done in
software.
And the way you make the thing run
faster, forward traffic faster
in software is to be able to say,
oh, that's in this address range. That is
a slash 24.
I don't need a subnet mask. I don't
need to do the XOR operation.
I need to do nothing. I can just
trim stuff off, and I know exactly where
to trim stuff off and ignore
because I know the address range, and I
know it's classful.
So people free, oh, that was so stupid.
No. Actually, at the time,
classful forwarding was really brilliant. It was elegant
and and compact. Yep. Yeah. And it made
forwarding fast.
I I I think it's interesting. We so
in order to make MPLS work,
we had to we had to we had
to virtualize forwarding. We had to create we
had to create an overlay. Right? Because because
of the constraints that we were working inside
of.
And
in in the in the act of creating
an overlay, even at the time it was
being made, I don't think people realize, even
the people building it didn't realize what the
future potential of the technology was. And Sure.
And now there are overlays everywhere.
And, like,
it it took it it took those constraints
and those requirements to kinda dig it out
of us, the this idea of an overlay.
And it's it's hugely important as a as
a concept, as a way of thinking, as
an architecture,
to be able to overlay. Yep. Yeah. Well,
we went from
virtual circuits
in tunnels
to virtual topologies
in overlays.
And that was a huge
mindset shift that MPLS
drove in our heads
about,
no.
Just because it's a tunnel doesn't mean it
has to look like a circuit.
It can be something other than a circuit
and still be a tunnel. I can totally
ignore the source address.
Wait. What? Right. You can do what? And
maybe have no knowledge of it. Yeah. Maybe
even have no knowledge of the source address.
I don't really care.
Is it part of this topology?
Yes. Happy happy joy joy forward away.
I don't care.
Right? And that's that's like a total mind
shift. And by the way, a lot of
people still don't get this.
They still equate tunnels.
That's a really interesting point. I'd never really
connected those dots. Pun completely intended, Tom.
And,
you know, I think there are people that
have other had other problems with other tunneling
protocols and conceptualizing. Like, you know, we've been
doing GRE,
IP and IP,
IPSec, you know, all sorts of other tunneling
mechanisms that it it does take time to
get your head around, you know, and and
that I think you're you're spot on. Right?
The whole like, getting the idea of underlay
versus overlay, that all happened around this time
as well. Yeah. It was driven by MPLS.
Absolutely driven by MPLS. So that's
even if you don't like the technology, even
if you're sitting out there listening to this
episode of the hedge and going,
why do I care about MPLS?
You should at least care
that it created a mind shift
in the way we see networks and the
way we build networks. It made
the entire
concept
of virtual a vertical modularization
possible in our heads. Yep.
And 10 points to your house. That's that's
an that's an important point. Very good. Yeah.
Yeah.
Good. So what else is there, Scott?
I've covered all the good things I can
think of. No.
Well, we, we have more to talk about.
I I I wonder what you you think,
both of you, on
I think around this time,
you know, the whole idea of separation of
a control plane and forwarding plane
was also
introduced,
which I think is at least partly motivated
by chipsets that could handle MPLS.
I, you know,
I spent a lot of time at Juniper,
so I have a very Juniper centric view
of this. I know Juniper did not have
a corner on the market of, you know,
separation of control plane and forwarding plane. I
think they did some other
things
from a router architecture perspective
that let the forwarding plane
forward
and the control plane, you know, handle all
the brainy stuff and then only send those
updates that are needed down to the forwarding
plane when routing topology changes actually
actually motivate that.
Yeah.
SDN started out really as a as an
MPLS technology or as an MPLS as as
we can use MPLS.
And so, yeah, there's a tie there,
between those things. And, again, another tie people
don't seem to recognize or remember,
but that is definitely there was definitely a
tie in that area.
And I think the one of the interesting
lessons too is so the control plane in
MPLS is twofold.
So so there's some sort of carrier, mostly
BGP if we're talking l three VPNs,
and then there's, then there's the actual label
assignment mechanism, mostly LDP,
in the in the case of fairly classical
l three VPNs.
Like and I know there are variations on
this but the the whole idea that you
have two different,
two different algorithms running independently,
doing different things but cooperating to the same
end.
You know, that's a that's a type and
shadow of the orchestration of today and and
it's architecturally correct to let different components do
different things and cooperate with each other that
it doesn't have to be a, you know,
a monolith.
And I think that's I mean, I think
that informs a lot of obviously, it informed
router architecture because if you have these two
things running as separate daemons that that's gonna
drive different, CPU choices and things like that.
But,
that's also I I feel like a like
a fundamental concept.
Well, to to that point, like,
I remember this
moment from twenty five years ago
that
you saw the convergence of how do I
get
RSVP
and BGP to work together?
And when
you know, so Junos by default, you know,
if the BGP next hop is the egress
point of the LSP,
Junos would automatically install
an LSP as the next hop to that
BGP endpoint. And I was like,
woah. It's magic. It just works automatically.
But, like, those operational things matter, right, to
bring the things together to people who don't
understand,
you know, or or have no desire and
need to understand all the protocol mechanisms underneath.
That makes me remember I was working on
a a pretty big MPLS network, and
we were trying to resolve some issue that
hadn't been fully baked before we started putting
configuration down. And and
somebody somebody was like, well, let's just next
top self it so we can go home.
Like Right. Right. Next top self,
facts come up, everything comes up, and traffic
starts flowing. And we're like,
oh.
It was like, yeah. We we should have
already known that. But
I, I wanna poke at Russ here on
something that I think will will get him
to respond.
You know, for all the
out there,
I love the ability
for, you know, BGP to do label distribution.
It's just another NLRI type.
And while some people will use phrases like
the garbage can of routing protocols,
looking looking at mister White,
I see that as elegance
and utility.
Like, the separation of protocol mechanics,
in BGP
versus what I can advertise with it,
dude, I just think that's super useful.
And I think, like, that's the polar opposite
of the mistake
of OSPF,
version two, right, where it could only forward
IPv4 addresses or only deal with IPv4.
A practical decision at the time,
but,
I'm in the I love BGP camp for
how flexible and accommodating it is. So I
would say two things to that. The first
is I still think BGP is the trash
garbage can of the Internet. We still do
way too much with it. I I it
only took till it only took till twenty
six minutes into the recording.
Yes.
But the other thing I'll say is part
of my issue
with the way we've done BGP, and this
started way, way early on,
is that BGP has two ways
to,
to segment data
or to
multiplex
data streams.
The first way is
BGP
has the ability to set up multiple sessions
to carry different kinds of information.
Mhmm.
The second way is is you can create
these address families that were designed originally
to say,
okay, this is IPv4, this is IPv6,
this is x and s. This
is IPX.
This is Vines. I mean, seriously, I mean,
there was all that stuff in there when
not that they they were never codified, but
that was, like, part of the thinking was
we're gonna have different kind. We we live
in a multi protocol world. We need to
be able to carry different address families.
And what we have done
is every time we want to add information
to BGP,
we stick it in as an address family.
Right. With all sorts of magic garbage
to backfill stuff.
And it's like,
no.
I can actually negotiate
in BGP
that I'm going to carry
IP reachability information
in an address family,
and I am going to carry MPLS labels
or security information or whatever I want
in this other stream, which has a completely
different TLV format
that is designed
for that kind of data.
And for some crazy reason, we decided to
make everything
an IP v four address,
like, somehow, like, treat them all like IP
four. Oh, we know how to use address
families. Let's do address families. Let's call starting
with what you know. Like, that's that's there's
a simple answer to that, Russ. Yeah. Come
on, man. But it but it but it
really broke.
If you go look at BGP implementations and
you look at the the switch statement
Right. That handles address families,
That is an extraordinarily
complex piece of code
that could have been utterly avoided
by just saying,
alright. This is an MPLS label. It is
not an address family.
This is different kind of information.
We're going to negotiate
a secondary
TLV or another TLV. Sure. And then, by
the way, you wouldn't have all this garbage
of, are they sending me
are they sending me MPLS labels?
Are they not? You know? What are they
sending me? No.
Did the session negotiate
MPLS
labels?
Yes or no? It's it's not that hard,
Russ. Right? Okay. What is what is the
NLRI type? And the NLRI type is the
discriminator of, like, what is the payload being
carried? So Yes. I know that. I'm not
backing I'm not backing down on this one.
I know that. But let me So let's
let's let's let's twist this. So
I actually had a couple really interesting conversations
last year
with Vint Cerf,
with Jack Haverty, who was,
really early on in EGP,
and then Lynn Bozak.
And,
like, Lynn talked about, you know, the getting
BGP
to be adopted by as many players as
possible
by router manufacturers that we would not even
recognize the names of anymore, Proteon,
Bay Networks, and so forth.
Stop it. Sorry.
They had to they had to be very
minimalistic in what was required.
So and that was, like, what eased adoption.
And I think that might be kind of
a fundamental contributor to some of the things
that you claim are problems or deficiencies.
But
going forward, like, so
we're pretty stuck on BGP,
like it or not, today.
It's not gonna last forever.
So by the way so by the way,
this Scott, this this whole thing about the
way we format stuff is something that Alvaro
and I thought,
like, ten years ago.
We actually wrote drafts Sure. Around,
please stop carrying this stuff as an NLRI.
Please start. There is this other way. We
can just negotiate a different message type.
And,
like, we don't need to put everything in
the same message type. And the community
explicitly
rejected it because,
well, what's the difference?
You know, we can carry
we can carry fake I p v four
addresses with communities if we just wanna carry
communities.
Right. And I'm like, no. Just
stop.
This is the difference between someone who is
writing and maintaining code and someone who is
writing and maintaining philosophy.
Well, there and and to the the point
where I'm trying to
sorry. I'm I'm not trying to sit in
the driver's seat on this. But,
you know, there comes a time when
incremental
adaptation
isn't gonna be sufficient and something new
Mhmm. Is more useful
and that has solid design principles underneath it.
What what comes next after BGP, and what's
it gonna take? And I don't know. Like,
I haven't I haven't thought about that much,
before five minutes ago. Again, yeah, we tried.
There was an attempt to make a a
follow on version of BGP, but
the community is is not interested in such
things at this point. Maybe in twenty years,
they'll be interested. There were several attempts, actually.
One of which was SOBGP
and past eight vectors, by the way.
There were others that were proposed by college
colleges to say, you know, research departments to
say, we could do better if we did
x. And so far, the community has said,
no thanks. Even if it's incrementally deployable,
which which is, okay. Fine. Whatever.
It is a heavy lift. Like, this this
it it's gonna take there have to be
significant value
in migrating to bgp. Next, you know, or
whatever it's called.
May maybe it's gonna,
you know
again, thirty two minutes into the episode,
you know, maybe AI is gonna figure out
a better way to do this.
And once we Oh, no. No. That's scary.
No. Please. Yeah?
Well,
I heard a rumor
that the AI was hallucinating
about the reachability
of a given destination.
Yeah. Sure. That's called a black hole.
And that's that's not artificial intelligence.
That's real stupidity.
So Gives us things to route routing by
rumor.
Routing by insanity.
So
I I I think, you know, there might
come a day when cars have less or
more than four wheels, but it's not today
and it's not within our lifetime. That's the
sort of where this falls in my opinion.
But Gotcha.
Yeah. Don't worry. I don't it doesn't keep
me up at night.
But there's a there's a really interesting guy
from Arista who is out there talking about,
you know, this this is a possibility.
And and look, I'm all for,
give me the right tools to optimize,
traffic flow in my network.
Right? And I don't care what those tools
are based on computationally
if they're reliable
and they do a better job more quickly
than a human does.
So, like, I'm all for that,
you know, again,
that criteria that I just stipulated.
Yeah. So going back to MPLS.
So we've talked about some of the reason
it was invented.
We we've almost touched on a little bit
about why people why people don't like it,
which rightly or wrongly, there are reasons that
people people consider it more complex. It's a
new forwarding plane. You know, if you look
at
the way we build networks,
and this is part of the problem with
v six as well. This is not an
MPLS alone problem. Sure.
We built networks
the way they were originally designed was
IP abstracts
all transports.
And that's not really that important to us
today because Ethernet's pretty universal.
But back in the day, we had ARCNET,
token bus, token ring. Right.
Aloha,
like, all of this stuff.
Stuff, frame relay. There was, you know, o
c three, d s ones, t ones. Like,
there were all these things, PPP,
all this stuff that you had to transport
on top of.
So having an application developer write to every
one of those
was an unhappy thing. Not it doesn't scale.
Yeah. It doesn't scale. So it's much happier
to have a single wasp waste or a
single point where everything comes to, which is
IP.
Yep. So one of the complexities
WASP waste. You are trying to replace the
thing
that was put in place to to
protect the transports
transports.
And so you're you're, like, in the middle
going, yeah. Well, this is the part that
makes the whole thing simple. It's like the
middle Jenga block. Right? It's like I'm gonna
pull the Jenga block out and put a
new one in.
Yeah. Well, okay. Good luck with that one.
And so that's part of the reason that
MPLS
struggled, I think, early on. And in fact,
it's part of the reason that IP v
six has struggled.
Another reason I think MPLS struggled
was because of the sheer complexity of label
swapping
Mhmm. And the label space not being unique.
Yeah. That is so hard for people to
get their head around. People don't even understand
net addresses in ISIS.
Right. Right. Like,
you're getting with the label? Yeah. I'm good
with NSAP addressing. You know, my ATM background.
We we can go there if you want.
But, like, I don't need I don't need
an end to end label value. Right? Like,
it's you know, I
the operation on the router
is, you know, read it, pop it,
you know,
push the new label on and send it
on its way. And, of course, you know,
the real value in MPLS is being able
to learn terms like penultimate hot popping.
That's just fun to say. So
Yeah. But I do substantively, let me talk
about the two things you just brought up.
Was is MPLS
does it add complexity on top of IP?
Absolutely.
Right?
But there is a generational thing on,
you know, okay, who's it complex for?
Right? And the older guys with the white
stuff in their beard and multiple grandchildren,
you know,
that was new stuff that we needed to
learn, figure out how this glues together.
Whereas people who came into the industry after
MPLS had is, you know, more widely adopted,
probably had less of a difficult time getting
their heads around it. I'm gonna I'm gonna
stick by that statement. So,
but let me also say
and and I'm gonna get in trouble with
some of the v six folks listening.
I should only be willing to increase complexity
when it adds value.
Yep. Absolutely.
Absolutely correct. Did MPLS
add value?
Yeah.
We also have to discriminate, okay, what were
really hardware advancements versus MPLS.
But I think you make a pretty clear
argument that
there was value added
and why it got adoption in carrier networks
and core networks
pretty quickly.
V six
is more complex,
and the value has not been
clear,
at least not in the early days. I
mean, when you get to address starvation, you
know, for v four, then you have a
real, you know, value add.
But the first twenty years
were like, I don't see the value of
of adding this complexity. Well, there's there's more
complexity than value in many cases.
And that's partially because
the community
decided,
for whatever reason,
that they weren't just gonna increase the address
space.
They were going to eliminate network address translation,
and they were going to eliminate fragmentation.
And they were going to do this, and
they were going to do that. And you
end up with this huge long list of
stuff that you're going to do,
and you end up with this much more
complex transport. We're gonna get rid of DHCP.
Why?
Right. Why does it work? Yes. It works.
Why are you getting rid of it? Sure.
You wanna add auto calculation?
Got it. Perfectly fine. Add it on. Make
it optional. Whatever you wanna do. Right. But
for for ten or fifteen years during the
entire development process, the v six folks were
saying no more DHCP.
We're not even gonna write DHCP for v
six. We don't like it. And, like,
who do you think is adopting that? Right.
Who do you think is gonna deploy that?
Well, this this is where I think the
comparison between v six and MPLS kinda gets
a little bit strained because in v six,
we ended up like, no. We're not gonna
do that. And then, yeah, I guess we
will. Like, in a lot of those things.
And so and that created, like, that
divergence of opinions created, I would say, complexity.
When you have to choose between
Slack and DHCP v six, it's a choice
nobody ever had to make before. Yeah. And
any anytime you force someone into a path
where, like, okay, now you have to choose.
Oh, now you have to choose again. Now
you have to choose again. Like, that is
real complexity. Yeah. To me, like, swapping out
the forwarding paradigm, that's not anything compared to
the complexity of the operators not gonna make
10,000 choices they never had to make before.
And I I don't know. I I think
the
probably conceptually MPLS was hard because of the,
like, the the label swapping idea that you
don't have a globally unique label space like
that really messes with with people and they're
like wait a minute. Overlay not a not
a tunnel. So then I think those are
the hard
things. Right. Right. And you have to you
have to think what you have to think
more like a router to really get your
head around MPLS, whereas you don't have to
think like a router if you're just doing
a pure IP. You can think like a
like a postal service employee or like a
regular human. Sure. And I think that's, you
know and
sometimes people just don't wanna think like routers
and that's okay. Like, I don't like, sometimes
the best operators are ones that don't think
like routers, I think. So I don't know.
I see there's,
I see the the pushback to it, but
but also,
I don't see like, I disagree myself with
the complexity argument.
I don't I don't see complexity in MPLS.
I see more I see a few more,
not that many, a few more moving parts,
but that stuff had to be in there
anyway. It was just hidden from your view,
where it's exposed to your view with MPLS.
Yes. I I appreciate that perspective.
I I do think I would say literally,
it does add complexity,
but not a
not an additional cognitive load that's impossible
or super, super hard to get. Right? Because
I if I figured it out, most network
engineers can figure it out. So
Yeah. I don't know about that, Scott. But
Yeah. Okay.
But there's one other thing that I think
really
caused a lot of angst around MPLS
and ended up driving it primarily
into the provider space.
And that is
just that some vendors decided
to charge extra
for an MPLS license Yeah. That's true. Which
makes no sense to me. It's never made
sense to me.
The hardware is already able to do the
job. You built
it into your ASIC. You built it into
your forwarding plane.
The code exists,
and then you are going to charge customers
to turn the code on.
At the same time, you're trying to get
customers to adopt this technology.
That
seems pathological.
I can see it though.
Just taking the other side of the argument,
if you're if you're thinking about support costs
from the vendor's perspective, not the user, the
vendor's perspective,
and you give all these free knobs to
someone,
what's gonna happen? People are going to turn
them.
And and people who did never never had
to make a conscious choice cost wise about
what do I do I purchase this feature
or not? And they had nobody had to
justify to anybody. Oh, yeah. MPLS is just
free. I can I can see the reasoning
from a from a support,
you know, a technical assistance
standpoint? Like, what's gonna happen to our load
when these clueless customers call and they turned
on label switching and now they're in this,
like, this this weird situation because they didn't
understand,
how much time are we gonna spend intact
trying to solve
problems from people who just they just didn't
know.
I I don't know. I I think that's
a little bit weak, but I but I
that's the first thing that comes to my
mind is a counterargument. I don't what do
you what do you guys think? So with
my with my tech engineer hat on,
I I I think there's something to what
you're saying, Tom. I don't think it's the
primary motivation. I'll come back to that.
I'll say that,
you know, as in in my tech seat,
every quarter,
we had training
on, you know, the next set of features
coming out in Junos.
And, like, it was some of the richest,
deepest training
that I ever had. It was engineering presenting
to TAC and other other groups within Juniper.
And that's just
of course, there's new features every quarter. And
there's a
you know, that's COGS, cost of goods and
services. Right?
So you just had to be up to
speed on, okay, what's coming in the next
release and be ready for it,
and then know the SMEs and the escalation
team,
you know, the the very, very deep MPLS
guy, the very, very deep ISIS guy, etcetera.
So
I I do think the bigger motivation
is the product managers and the marketers trying
to find ways to monetize the value that
they're bringing to the market.
Rightly or wrongly, I'm stating this without judgment.
We may save judgment for later, or maybe
Russ will take care of all of that.
I I do understand the need to, like
okay. We we're doing really important work that's
enabling value in customer networks.
Why don't we find a way
to capture some of that value as well?
I I think there are okay ways to
do that. I think there are stupid ways
to do that. Yeah. And I'll my my
favorite example of the stupid ways to do
that
was in the initial,
j series, you know, small CPU routers from
Juniper.
Yeah. You had a two port t one
card,
and you needed to buy a license to
activate the second t one port.
Are you kidding me? Yeah.
That's that's a bad example. I'm sure Russ
has other examples. No. No. I'm I I
think MPLS is a bad example of it
as well. Like,
it's in the soft it's in the hardware.
Like, let it go. Now, you know, by
the way, when I was in tech,
we often wanted to convince
the the product guys
to ship all of our routers in Chinese
puzzle boxes
or or
locking
or
or locking nesting dolls, you know, because, like,
if you ain't smart enough to open the
box,
please don't be calling TAC. Okay?
That's that's nice.
And that's why TAC engineers don't have a
direct line to be good product management.
Right. Number of times, like, we used to
get phone calls all the time.
Well, my router came with a disk, a
three and a half inch floppy disk,
but there doesn't seem to be a disk
drive. Do I have to buy the disk
drive separately?
Right. And you're like, I I don't even
know where to start. I don't even know
where to start.
I hear
you. Oh, Scott. I like your example of
the the t one is actually exactly what
MPLS is. Like, the hardware is there. It
can move the bits. Yep. It's, you know,
r and d has been done and paid
for. Like, there is there is no, like
but but then, you know, the argument to
that could be, well,
why and this leads to, like, pay as
you go and leads to all sorts of
licensing schemes which are meddling. Totally. Totally. But,
like, we'll do we'll we just won't make
anybody pay for it if they don't use
it, and that's better for everybody. Right? It's
like, no. No. I
Yeah. Well, we do the same thing right
with routing protocols. Right? You buy this version
of iOS and you get EIGRP.
You get that buy that version to get
IS to IS. Like, what
I you know, I don't know. Yeah. Like,
I don't understand.
So that was a,
you know, a a
again, you know, on the Juniper versus Cisco
days. Right? Early on, I would say the
first ten
twelve, thirteen years at Juniper,
we were able with a straight face to
say, look, it's all in there and you
don't need to license it. It's it's on.
And we that that was a great competitive
value Yeah. Against against Cisco.
Now that
static, you know, over time.
Product management
does find a way to, like, okay. Maybe
we need you know, engineering will say we
need different builds because this code's getting too
bloated, and then I have to license certain
features. And so and that's how new companies
arise.
Yes. Like Arista. Like Arista. Yes. Exactly.
Yes. It it it it really is.
I don't know. And and so, by the
way, this is the difference between the companies.
When I say this, people like, what do
you mean? Cisco is a marketing driven company.
Yeah. It just is. Always has been.
Juniper
was always an engineering driven company,
and it always was. And so Juniper could
never market themselves out of a wet paper
bag.
And Cisco
had really brilliant engineers, but the engineers were
stuck in some little corner in in a
darkened hallway,
and the marketing people were determining everything that
should happen.
And so, you know, it's just different company
philosophies.
You know? It's just different different ways of
looking at the problem of making money.
And so, yeah, I don't know. It's funny.
So I've always I've care like, Juniper, I've
characterized exactly the same way.
Cisco, I would have said, is a sales
driven company. A sales marketing. Yeah. And yeah.
So those are adjacent enough. The Venn diagram
has a large intersection.
Yeah. And I think, you know, you look
at John Chambers and, you know, having been
a sales guy into the CEO role for
so long,
this shouldn't be a big surprise. Right? Yeah.
No. It shouldn't be. Yeah. And Jennifer was
led by engineers. Right? I don't
yeah. But, you know, that's Up to a
time. Right? And that those things change over
time. So Yeah.
And and, like, there are Nokia comparisons here
too just from, you know, my time there.
I would also say they were engineering
driven.
How do I say this? I'll just I'll
just be direct.
I never dealt with a more technical
and business savvy centers
where
I
I knew that I was not gonna get
a feature request implemented
unless I show that I knew what I
was talking about and could really justify it
and had aligned to 10,000,000 NPOs.
That's an arbitrary number. But there was tech
technical qualification and business qualification.
And
I always felt, you know, like, having spent
time at
Cisco, Juniper and Nokia,
that helped Nokia
deliver
code on time with quality
in ways that Juniper and Cisco really struggled
with.
Yeah.
So, yeah, corporate corporate culture
in Cisco, if the salespeople say I have
$250,000,000
deal on the table, the code gets done.
Yeah. Regardless.
And at Juniper,
if an engineer says a a distinguished engineer
says I have a brilliant idea, the code
gets done.
Right. That's the way it works.
And so yeah. So it's it's an interesting
but, anyway, back to MPLS. So those are
I think those are the biggest arguments against
it or the complexity.
To me, that you're pulling the Jenga block.
You're trying to pull the the central Jenga
block out, and that just makes it, like,
that's a mental
that's a mental thing for people.
And
then the cost. I think that, you know,
people charging more.
Have we overcome those? Are people now
beyond all of that and deploying MPLS freely?
I think the problem is in most enterprise
network, it's got gotten such a bad rap
that nobody even looks at it anymore.
Yeah.
Except for maybe the enterprises that operate,
like, their own core network.
Mhmm.
But I would take I would take that
comment and flip it, you know, for the
enterprise users.
Maybe I don't even need to care about
MPLS.
I don't care, you know,
Verizon, AT and T, Comcast, Charter, what you're
doing in the core network.
Just meet my
don't care how you do it.
Don't care how you market it. Right? And
if SD WAN can do that for me,
I'm okay with that too as long as
my requirements are met.
I think this it's the scope of the
problem you're trying to solve is really critical
here because a small scope, almost always
almost always excludes
like it well, I should say a small
scope makes it easy to exclude MPLS if
that's what you'd like to do as an
engineer.
But there are some problems that you you
can't honestly say, well, I'll just push it
to this other part of the network. Like,
the one that I'm thinking of in my
experience is,
when when you have to do traffic engineering
because of because of latency and when you
when you have a global network, you can't
you can't just say I'll throw more bandwidth
at the problem because latency doesn't doesn't respond
to bandwidth. Right? So, like It's a sad
light hop. That's, you know
Right. Right.
Light hop popping. But, like,
but if you so if you're doing business
in Shanghai and you're doing business all over
the world,
and you need and you need packets to
go a certain way and then you need
them to fail over in a certain way,
you can't just say, well, I'd I'll just
throw bandwidth at the problem. Like, that's that's
not a solution. There's not a like, the
eventually, you do have to deal with an
overlay. You do have to deal with label
switching. And this label switching isn't the only
solution, but it's like there are there are
some problems of such scope that you you
don't get to just arbitrarily shift the complexity
somewhere else. You have to actually look at
the architecture,
from the ground up. And I think that's
where that's where MPLS shines when you have
when you've got these problems that are fairly
intractable with other things and you've run out
of other options, then,
it becomes really good. And then I think
when you work on those networks, at least
for me, when you work on those networks,
you really start to appreciate,
how
how how,
elegant all of this is that you can
actually design a network that spans the entire
world that meets your requirements for latency and
and is survivable.
Like, that's impressive that you can actually do
that. And,
and to do that, you just have to
have the the right tool for the job.
Yep. Totally agree.
Yep.
Yeah. So sometimes you just have to eat
the complexity or the perceived complexity whether or
not it's complexity is another question. But,
just to get a certain job done, hard
problems require complex solutions.
Yep. They just do.
You know, so sometimes complexity is not bad.
It doesn't mean you shouldn't contain it. Doesn't
mean you shouldn't separate it from other complexity.
Doesn't mean you shouldn't manage it.
Just means it has to exist.
It just is what it is. It has
to exist.
Yep. Yeah.
Can I throw one other final hopeful comment
in on that?
I
I think, you know, we've
we've seen so in our own careers and
so many of our our colleagues and friends
develop subject matter expertise in certain domains.
Right? And we got limits. We got limits
on the amount of complexity that we can
process. Maybe Russ doesn't have any limits, but
No. No. No. No. We have limits. Trust
me. We have limits. And and one of
the things I am
curiously hopeful about
is the ability of power tooling that comes
from the AI world
to span technical domains,
you know, even within networking, like what's happening
at my optical layer, what's happening with IP,
maybe even bringing in data from the security
world, and maybe even bringing application performance data
in, and take a more holistic view of
the problem.
I'm
you know, it's not magic, and
I'm
not saying this the way that means that
that that implies you should go get another
job in a year. That's not what I'm
saying.
But I think there's gonna be ability
to span those technology domains
and come up with more holistic solutions and
approaches
than I think we've ever been able to
do in the past, and that's pretty exciting
to me.
Yeah. I I like thinking about it like
that. I think there's,
I think human judgment is is really amazing.
And the thing that sometimes our our judgment
isn't we're not able to employ it fully
because we lack domain expertise in some areas.
And so we we end up in our
our own, like you're saying, our own our
own domain expertise.
But if you could connect domain expertise of
of multiple people or if you could, you
know, basically, if you could bring your human
judgment, which I a AI will never replace.
Correct. And AI doesn't have the ability to
supplant.
Like, if you could bring your human judgment,
which is such a powerful machine
closer to subject matter expertise,
then I think you could do a lot
of things. Like, I I I a lot
of the value of AI, I think, could
be explaining to someone, like explaining to a
person who has x x domain expertise.
Here's here's the actual problem you're trying to
solve. Yep. Now what do you think human?
You solve it. I'm just a dumb machine
that guesses the next word in my large
language model. That's that's really pretty stupid but
but the human judgment can really, I think,
be augmented.
Yeah. And what we're actually talking about here
is the first corollary to Keith's Law, by
the way, in case you wanna put
a a formal
A label on it? Did he A label
on
it.
That's the best way to end the episode.
Sorry.
Which is that you can know you're a
part of the network or that you're part
of the system really, really well. And you
can know the pieces that are adjacent
fairly well.
Beyond that, it's all rumor and pop psychology.
Yep. I like it. Yep. We got limits.
We got limits. We have limits.
Okay. Cool. Alright. Anything else you wanna talk
about with MPLS, Scott, before we
go away? We don't have all day.
I I could continue. But,
Well, we could we could schedule another five
episodes
and make Whatever.
If you really need to fill up slots
that badly, Russ, I'm happy now.
Yeah. Look, it's been an interesting
technologies to watch over a career.
And,
you know, for all of us, it's a
snapshot in time. There will be something next.
Right? You know,
whether it's gonna be, you know, whatever supplants
BGP,
what's beyond IP even,
won't speculate. But, I feel pretty fortunate to
have had the seat at the table that
I've had and just to see the pieces
of it. And and to be able to
hear from, you know,
Tom and Russ on stuff like this, it's,
it's a great spot to be in.
Yeah. I think there's a big
part
of being a network engineer.
Like, when I tell people, when I'm teaching
CLO fabrics and I say, I actually used
to work on a stroger fabric, and they're
like, wow.
That's, like, crazy
that you actually worked on a stroger fabric.
Well, yeah.
But, like, that's part of understanding it that
I don't know how we replicate
in a younger generation,
unfortunately.
It's just the the physical experience
of taking that can of contact cleaner and
going over to that
one in, 100 out stepper motor driven
stroger switch,
spraying all the contact. Somebody called in and
said, my connection to the base commander was
fuzzy today. Do you think you could fix
it? Or, you know,
somebody's gotta do it. Yep. Somebody's gotta do
it. It's like it was, you know, it's
crazy.
So anyway alright. Tom, anything else on MPLS?
Mostly pointless label switching, nipples.
No. But I'm I'm working on another another
mostly pointless. It's just gonna be another clever
thing we can we can use those letters
for. So
MPLS,
the city in which MPLS
was Born? Born or moved forward or however
you wanna put it, hummed into existence or
whatever you wanna say. Yeah.
Cool. Alright.
Scott, where can people find you if they
want to or follow you? Check check out
LinkedIn.
There aren't too many Robon's, and there's only
one Scott Robon that I've ever come across.
And you can see other stuff we're doing
at solutional.com.
Reach out anytime.
Cool. Tom, say your word.
LinkedIn.
Audrey made him wait a long time that
time.
Yep.
Alright. I'm Russ White. You can find me
here at the hedge, rule11.tech.
I don't know. I feel sometimes I feel
like banana man. He's everywhere. He's everywhere.
We know that.
We live in an attention driven
and so your attention is extremely important. We
thank you for listening all the way to
the bitter end of this hedge, and we
will cut catch you next time.