Join us as we gather around the hedge,
where we dig into technology, business, and culture
with the finest minds in computer networking.
Well, hello, Audrey
and Tom.
Audrey says hello. Audrey is the plant, you
guys.
Yvonne just named just named her last time.
So Yes. Now we have to use Audrey.
So, Tom, how are you this morning? I'm
great. Getting, getting ready for the freezing cold
to hit Texas. We're not ready.
Oh, your rocket is upside down by the
way. If you continue to allow it to
go down Oh.
Yes. It doesn't it doesn't it doesn't work
that way, Tom. Okay. Okay. It over. I'm
glad you told me.
I wouldn't I wouldn't want bad things to
happen to your house.
And today, we are joined by Doug Madery
from Kintek
who is in the woods.
Hey. How's it going?
I mean, I just guessing from your background,
you're out in the woods in a tent.
I'm in my orange tent. Yes. You're in
your orange tent.
So there you go. And then we have
Job. Job, where are you physically?
I'm in Amsterdam at the moment. Thank you
for having me.
Cool. I actually love Amsterdam. Oh, and I
should clarify. Amsterdam
is a relatively large city in the beautiful
country of The Netherlands.
I, I have to explain because once I
was in a a jeans store and I
told the this little person I was from
Amsterdam and they responded that they they loved
Germany. And I you know, I never seen
it in that moment. Wow. I feel like,
you know, not everybody knows Hope you just
walked right out. Yeah. Yeah.
Wow. That's wild.
The last time I was in Amsterdam, I
don't know if they still have it, but
they had a Van Gogh exhibit at the
main museum there in town. And that was
really that was awesome.
Going to go look at the Van Gogh's.
Yeah. Yeah. Yeah. We have a Van Gogh
Museum, a permanent one, and it it is
gorgeous. So if you come visit Amsterdam,
come to museums. Yes. Yeah. Yeah. The museums
are great.
So alright. So let's talk about AS sets.
For those of you who don't know what
it is, an AS set is basically when
you aggregate routes in BGP,
you take all of the b g of
the AS path elements
and you combine them into an AS set,
and it becomes a single entry in your
AS path.
This may sound strange. Wait a second. That
is a difference
AS set that I thought we were gonna
talk about. Oh, okay. Well, then you then
you define it.
Alright. That was that was a plan. That
was that was planned. That was planned. Yeah.
We walked right into that, Doug.
So so, Ross, I think you are talking
about
a s underscore
set. Yes. That's
correct. Mentally
prepared to talk about a s
dash set. Well, you talk about a s
dash sets. That's fine. You go right ahead.
Another another instance in the Internet routing terminology
where you've got overloaded terms in a way
that is problematic about Router switch
aggregate Well, I've been talking about this in
the last year. I have a slide right
up front. I was like, this is a
talk about AS sets. If you came for
a talk about AS sets, you're in the
wrong place. This is just AS sets. I'm
not gonna be covering. So the the same
term, you know, whether it's dash or underscore
refers to two different things.
I think the thing that, the the route
consolidation
piece is a is something that's,
been deprecated.
Although, it's still like, everything that's been deprecated,
there's some traces of it out in the
out in the wild. Yes. Correct. Is the
ASF mechanism to try to,
create kind of allow lists that you would
allow what prefixes you would see over a
BGP session,
in the hopes of trying to prevent
Okay. Well,
that let's start there then. So explain to
us where how people so this is an
IRR
application,
right, or an I IRR
application.
Correct?
Right.
IRR.
Incident
routing registry.
I always say them backwards, so don't worry
about me. That's okay.
And
IR is is a mechanism.
If you trace Beckett's history,
you you end up at the
late eighties, early nineties
when,
super clever people had the foresight that in
order for the Internet to be able to
grow,
we would need
some automation
here and there.
And this automation,
these pipelines would need to extend across organizational
boundaries.
So,
people realize, like, hey. I need to configure
my equipment in a certain way, but I
should also inform the world about how I
am about to configure my equipment or how
my equipment is configured.
And then others
based on that information can,
generate configurations
or or make routing decisions.
So,
IR is is one of the
the earliest attempts at Internet skill coordination,
in order to to to grow the network.
So so when we when we talk about
things. When we talk about configuration,
what we're actually talking about is policy.
We're not talking about, like, network statements or
blah blah blah. We're talking about who transits
where, who's allowed to transit, for what reason,
who who's peered with who. That's the kind
of policy we're talking about. Right? We talk
about configuration for IRRs. Like, these are my
upstreams.
If it's not from my upstream, don't trust
it kind of kind of policy. Is that
a correct?
Yeah. Yeah. It's it's,
that's that's one example. And and now we
almost immediately hit upon one of the challenges.
IRR is super flexible. So not only can
you
try and express who your upstreams are, but
maybe also who your downstreams are or your
neighbors,
of a particular type. Or and now things
get kind of funky.
You could make a list of odd numbered
ASNs
just because you feel like that is important
for you to share with the world.
And
Doc may make a list of ASNs that,
you know, he he just likes the shape
of their numbers.
So what an ASF
is
is is up to the creator,
and everybody
sees something different in them. So this is
immediate like,
it would be cool if the application is
you can list your upstreams, but that's not
technically
what
it always means. It it could be upstreams
or downstreams or something else. It could it
could be your summary peers. It could be
Yeah. It could be the peers you pay.
It could be private peering spy. It could
be anything. It's just It could be anything.
And and in computer technology,
if the moment somebody's like, it could be
anything, it's like, it will be everything and
more. Yeah. It's where trouble starts.
Maybe I'll maybe I'll mention, well, I think
we'll circle back to these issues that, Joe
was touching on. So what the way I
got started on this,
analysis was looking at routing leaks, something that
I look at mishaps, routing mishaps, and, and
noticed that I was aware of this issue
of the AS sets, for a a number
of years where there are some
networks out there that that have the AS
set or one of the AS sets that
they or all the AS sets that they
publish to the Internet
are humongous,
so big that they encompass basically the entire
Internet.
And what this is intended to be is
a white list that you're or allow list.
That's a,
not a a technical term. It's a conceptual
term of just what would you what prefixes
should you be allowing across a speed to
be session. But when these things expand to
be,
you know, expanding to be the entire Internet,
well, now you've just it's permanent any any,
and,
and it's not doing anything. It actually and
then it's not doing anything, but it it's
even worse than that because it it's costly.
It's done in a way that's that's very
costly on,
you know, the routers trying to expand these
definitions to be millions of lines long,
and white you know, white listing the entire
Internet. Anyway so I guess I I was
looking at routing leaks. It had happened a
few times where, the the leak was going
through, a network,
that had a an asset that you could
you can look it up and see it's
it's just way too big. It's hard to
with in a routing leak, it's hard to
be certain because there's so many factors that
influence
the propagation of leak routes,
and many of these I can't know. I
can't be inside the router to know the
policy.
But I do know that this is not
helping, at least, when the, the leaker has
a,
an ASF that's
massive. And and so if any of its
upstreams were using that ASF,
they wouldn't be able to have stopped this
adjacency leak.
And so in the in the also just
another clarification. On the in the topic of
leaks, we've got
origination leaks. So an AS, like, an announces
appears as if it was itself in AS
path. So right most AS.
And so we have a mechanism,
that,
is in a is in relatively good place,
ROV,
route origin validation. So that just that just
handles misoriginations.
Mostly for this is really for accidental,
incidents because, you know, one could add an
AS to AS path. We wanna defeat this
thing, but it's really for just trying to
shore up all the,
accidental misoriginations
that that occur on the Internet. I think
it's doing doing a good job. Now the
other issue is these adjacency leaks. You have
there's no new origination. It's just browser crossing
edges that they're not supposed to. That's where
AS sets, would,
potentially,
have a role to play. But like I
said, so I started to see these AS,
AS sets that were involved in, leaks,
and that were massive. And I was like,
just what how big is this problem? So
I started to go down the road and
just like, alright. What are all the ASFs,
and can I expand them, and how big
are they? I started to do that, and
I noticed that bgb.tools,
this is a neat free utility, kinda like
bgb.h.net,
has a a a mechanism that you can
explore all these AS sets. And I and
after clicking around for a little bit, I
realized, like, this has already been done. I
don't need to write this code. So I
reached out to Ben Carre Cox, who's the
founder of that tool of that, company. And,
I was like, I think you've already done
this, work. Would you mind just what what's
the what's the big list or what's the
worst? As we made this list of the
worst s,
assets
and,
and it's absurd. I mean, the the top
20 all have over a 100,000
ASs. So an AS set will expand to
ASs,
ultimately, and then those ASs get turned into
prefixes based on what they're originating
in as far as the IRR knows.
And the,
and so there are over a 100,000. There's
only, like, 80,000 ish,
ASs in the global routing table. So this
is already, like, problematic.
It's bigger it's bigger than everything that's in,
in the routing
table. And so, yeah, I started to compile
that and
So so let's so let's pick up a
second and
how would people implement this? If you were
an operator
and you wanted to use the IRR as
a asset capability,
Is there something built into routers? I mean,
there is, but okay. So can we explain
that a little bit about how you would
implement this so that operators can kind of
in their head, people who've never done this
before can kind of understand
how this would be used.
Yeah. Let's let's talk a little bit about,
like, the the happy flow, how AS sets
normally are intended to be used and why
why they existed. The happy flow. I like
that. The happy flow. Integrating. So,
Ross, you and me have a BGP session.
I am your supplier, your vendor, your transit
provider, and you are my customer. The goal
is that you announce your routes to me,
and I propagate them to all my neighbors.
Now
every time
you sign up a new customer who has
their own prefix,
if you have to send me an email
like, hey, Joe. There's a new downstream behind
me.
After some time, we both might find it
tedious and even worse, error prone to error.
For instance, you you forgot to email me
or I I missed your email, and then
then I didn't update the configuration on my
side to add your your extra customer to
the allow list.
So instead of that manual workflow,
the idea of
IRR and AS sets is that at provisioning
time, you tell me, Joe, this is my
AS set.
Periodically
expand this AS set. Expand it by using
tooling like BGP q four. Mhmm.
And then whatever comes out of that extension
is something you can load into the router.
And the goal of loading
some configuration into the router is to limit
the blast radius
of,
route leaks if they occur.
So the idea is that you know
who your customers are, what you intend to
announce in the normal course of operations.
And if if there's some kind of mistake
somewhere in the path, maybe a customer of
yours or or maybe your own organization,
has a typo,
accident.
If you inadvertently announce more routes than you
should announce to me, then the filter derived
from the AS set,
should should hamper the propagation or maybe even,
stop
the leap right in its tracks.
Now
the idea of an AS set is that
it's,
it has a recursive property. So you create
AS set
ROS,
and you reference in your ASR,
maybe Doug's ASR.
And then Doug, in turn, can,
reference his customers. And the idea here here
is that that you
don't have manual work all up the chain
where
the the entire supply chain is telling each
other, hey. I I got a new customer
or customer was removed,
but instead automate some of this.
But this is also our challenge to start
because
once we automate the creation of,
routing security configuration
on my side
and I blindly ingest whatever you put in
your AS set,
yeah, what's to stop you from misconfiguring
your AS set object.
And misconfigurations
are
way too easy in this context.
This is because there there are,
I'd argue,
information architecture issues with with the IRR.
What I mean with that is
AS
said ROS.
If there are multiple databases,
which of these databases
contains the authoritative copy, the one that
you, Ross White, famous from this podcast,
created?
Because there's multiple people on the planet named
ROS. Right? So maybe I think I'm pulling
an AS set Ross that it's by by
an entirely different Ross. It's Ross Housley.
That's fine. Right. Right. Yeah. So so pulling
in the wrong AS set obviously leads to
generating the wrong configuration.
So then the the purpose of the exercise
is somewhat defeated.
And I could be pulling in the wrong
AS set, but you might be referencing the
wrong AS. Maybe you intended to reference
doc Midori's
AS set, but instead, you get doc
Homer. Last name. Yeah. Homer Yeah. AS set.
So do AS sets with the same name
can exist in different databases and have different
content,
because it's it's not a globally unique namespace.
So that is one major major quality issue,
in trying to use AS sets. And
you might ask, like, why why are we
even using AS sets? Because,
well, at the time,
we didn't really have anything better.
And you you, you know, you work with
the tools you have. And if all you
have is a a blunt hammer, then, you
know, you you start hitting everything. And there
were and there were tools built around this
to expand these AS sets
into
act prefix list, not prefix, AS list
for AS prefix list for Cisco and Juniper
and Huawei and blah blah blah blah blah.
And in fact, I think there are some
tools built in to operate in network operating
systems and the BGP implementations
that will actually read data in particular formats
to do this.
And so it kinda became an ecosystem.
It's not just like it's a single thing
that somebody hacked out.
It became an entire ecosystem
that we began to trust
to solve certain problems,
in the global Internet
as a as a way of automating.
And so yeah. And so part of the
problem is gonna be, of course, just that
IRRs themselves are voluntary.
It's whatever stuff you stuff in
comes out the other side.
If you stuff the wrong stuff in, guess
what? The wrong stuff comes out the other
side.
And then, like you said, it's not globally
unique. There's no global uniqueness
to
to the actual data that's being carried there.
It's, One one one thing I might add
on this is that, just on the topic
of,
every all the points Joe is raising are
are great,
just
on on the idea of these things just
expanding without end.
As we as we kinda discussed, there's there's
there's always been a lack of semantics around
this of, like, you would use this type
of ASR. Some some ASR sets, you can
just read them by their name. This is
a ASR set of all the ASR's at
some IXP. Like, okay. Well, that might have
some particular use. There's another ASR. These are
all the peers of some other, you know,
ASR. Okay. That's that's that's could be useful
in some scenario. Here's the all the customer,
the customer code of some AS. Okay. That's
useful.
What you find when you start digging into
this
is that,
due to the recursion, you have an AS
set that has,
all the customers within within that
is all the ASs of an IXP.
And then and then again, all the I
I the, ASs of a of a of
a of a peer of one of the
members of the ISP at the IXP. So,
what ends up happening is, this system that
was built
to, police and stop routing leaks is itself
experiencing
leaks,
that are very akin to like, if you
were at a an AS at an exchange
and you took all the routes that you
learned at that exchange
sent them through your transit provider, well, that
would probably be a leak and you'd be
inadvertently providing transit for everybody at the exchange.
So that so we have the exact same
thing as happening within the AS set world
where these things get,
you know, rolled into other, types and the
semantics are not well established or understood.
It's,
I know Joe's got opinions on this of,
like, it's almost a a state where, like,
it's not clear. You could you could say,
like, oh, you need the ASPA or some
kind of another mechanism just for the AS
sets. You know, that was that's never gonna
happen. That's I'm not suggesting that. But,
it it may be unfixable, and we have
to have different mechanisms to,
to
police these kind of adjacency leaks. So is
there any is there any kind of I'm
assuming the answer is no. But is there
any kind of semantic for do not recurse
or, like, I don't wanna be involved in
this ever expanding web of everything? Is there
any way to express that?
No. And actually that I I I when
I last year, I was giving, you know,
a talk along these lines a few times.
I I'd get operators at the end, share
their, you know, experiences with this. I had
one that was a a guy who handles
this for a major European carrier.
And one of the things he he tried
to do was go into every AS set
where his AS appears and shouldn't and see
if he can reach the person there. And
just, like, hey. Would you mind removing it
and maybe tell me why?
Most times, he couldn't reach anybody.
The other and the other popular,
outcome
was that he would get a response back
saying, oh, that, the guy who made that
left, like, ten years ago.
And,
and and you're like, well, can you remove
it? This actually has operational impact on the
Internet. And you're like like, we don't even
know how how we would go about doing
that. So, like, this is We don't know
where the password is.
Exactly.
So,
yeah. That I mean, I don't know. We've
already named enumerated maybe a dozen issues here.
But,
so now another issue.
The feedback loop in all of this is
horrendous.
So let's let's take a VGP routing.
If that happens, then the circuits that is
affected by the route leak will probably start
running hot because your leak
is pulling in traffic that that shouldn't be
on that path. So,
the signals you have are, hey. There's congestion,
on the path. There the the monitoring shows
that interface is, 100% utilized.
Maybe there's a counter that shows that you're
receiving way more routes than you usually receive
from the peer. So these are all, I
would say,
fairly direct
indicators that something is wrong.
But in the IR ecosystem,
if you add a reference to the wrong
AS set or the or there's,
duplicate names for different
AS set objects,
there's not really a warning system
that materially
affects your operation like a congested circuit would.
So for most people, they will just
add to the AS set until whatever
task they had in mind is done.
But removing information
is a source of great uncertainty because who
knows
what the purpose of that information was and
why But, Joe, it's even it's even worse.
It's even worse than that. Let's say you're
making your ASF and it is airtight. You
this is the best ASF in the world
on the Internet.
And tomorrow,
one of the members of that ASF adds
something ridiculous and you just Yeah. Your thing
just went to crap, and you have no
idea because you what you're looking at is
the same as what it was yesterday when
it was great. And and and that only
is seven layers away or something. Yeah. But
the recursion can also be circular,
and the recursion algorithms
are not really standardized,
or deterministic because
one operator might use IRR databases a, b,
and c, and another operator might use IRR
databases
b, c, and d so that, you know,
there's if you imagine a Venn diagram that
there's partial overlap.
And if there's duplicity of name naming collisions,
different contents with different databases, and the databases
are
looped through in a different order, you know,
one an American operator might prefer to first
lookups
in Aaron's database,
whereas European oriented operators might,
configure their system to first do lookups in
Ripe's database,
you get very different results.
So debugging this like, hey, Doug. Are you
seeing this issue?
It's really challenging because we might be looking
at entirely
different
data structures,
and confusingly,
there might be duplicity of names. So So
is there is there anybody who's just
opted out? Are there any operators that are
just like, it's not worth the mess? We're
just not even gonna use this information anymore.
I got so what what before I was
getting to I was starting to do a
little research to to,
write this, analysis last year, I talked to
a bunch of,
kinda like what what we consider tier one
operators, cloud,
operators. Hey. What how are you guys is
this a problem for you? I guess it
was my first question. They're like, oh, yeah.
Like, we have to we basically had to
make another system,
that,
that runs over this. So people have had
to independently
solve
around this. It can't be used as as
is,
and so,
Google's got one solution. AWS got somebody else,
and everybody's kind of written a bunch of
software to try to take the sorry state
and then try to use some other heuristics
to pair it down.
Some things that are that are difficult are
there's like like, everything in the Internet, there's
all these corner cases. So you can have,
like, let's say,
if your approach bay is based on, hey.
What was routed yesterday?
Let's just assume that's probably what we're gonna
see today,
or maybe an hour ago. Let's assume it's,
you know, let's say it's even
timely. You've got, like, a few different weird
things like,
b g b b g b based DDoS
mitigation that could conceivably be announcing,
a lot of a lot of different things
on the Internet,
that won't wouldn't be there an hour ago.
But when it when it kicks on, it
it really needs to have those routes propagated
quickly.
So,
it's it's hard to make, I guess that's
true for any security mechanism to try to
deal with that scenario. But,
yeah. Every I think everyone that I I've
I've spoken to has had to write software
to So nobody nobody's
nobody's abandoned it per se. They've just tried
to fix it internally.
And and by the way, the one of
the problems you have here as an operator
is the bigger your footprint is in terms
of peering, the better your own internal solutions
for this are.
So smaller operators like Google's
are,
give off a false sense of security,
unfortunately. Yeah. Well, that's gonna be true too.
Yeah.
So you you have this system. It
takes this input,
as parameter,
the name of an ASR. It expands it
through some algorithm
into a list of prefixes.
Let's say a list of, 50,000
prefixes. Which by the way is not guaranteed
either. The entire idea that
anyway, continue. But that that's another entire problem
It's like this correlation between prefixes and AS
numbers is not as Right.
So so through some
non deterministic
process, it generates a blob of config that
is then uploaded into your routing equipment.
But this may give a false sense of
security because it might be the wrong
blob.
And
as these blobs become larger and larger because
of,
circular,
recursive,
references
or references
to wrongly named objects,
you may be like, wow. My configuration is
10 megabytes.
It must be super secure.
But the opposite is happening. What what happens
is you you made a,
a door that is wide open,
but it is obfuscated by,
the
the the sheer size of the the configuration
that was generated.
And it's not actually blocking anything anymore because
it's it's it became a really expensive allow
any any.
And
if you think of the firewall
policy term, allow any any, it's one sentence,
three words. You know? It's a very short
way
of describing,
what you want to pass through. I mean,
everything.
But imagine you could also make a list
of 4,000,000,000
entries with the 4,000,000,000 IP four addresses and
look try and load that into your router.
You'd be like, look. It's super secure. My
firewall filter is 4,000,000,000 entries.
And then so at some point, somebody's gonna
come along and be like, you know, Jill,
this is actually it's, you know, it's very
expensive to maintain, but it's not secure because
you
you you tricked yourself. It doesn't allow any
any and not a
allow a specific subset that complies with my
business,
requirements.
So this is a challenge with IR because
when we talk with operators, they're like, hey.
But I'm I'm doing the right thing right.
I,
the MANRRS initiative is encouraging me to filter
the routes, and I've generated giant filters and
uploaded those to my routers.
What am I doing wrong? Why are you
unhappy?
And then you have to go and explain,
like, well, it's it's it's a garbage in
garbage out system. So you've pulled garbage in
that looks
legit, but you have no
way of knowing where the data originated. There's
no
cryptographic signatures
on the data itself.
And you may be downloading this via FTP.
Okay. Then it's not secure.
But and maybe you download it through HTTPS.
Great. But that's transport security. That doesn't tell
us anything
about whether
that AS set was really created by Ross
or Doug,
or someone else.
So the data is extremely problematic to deal
with, but it it gives off these vibes
of, you know, if you work with this
data, you're doing something right.
And persuading people to take a step back
and let go of, you know, practices that
they've been following for twenty plus years
is not always easy.
Yeah. So so my baby ugly.
Yes. Sorry. Yeah. If it's IRR based, it
is not a pretty baby.
So so, Doug,
you would you say that
the origin
AS number and and and
prefixes are fairly solid.
That's not been my experience when looking at
the table.
It's solid enough that it gives you the
same sense that, like, Joe was talking about.
Like, oh, look. Yes. 99%
of this time, this prefix comes from this
particular AS number,
but that doesn't necessarily
Okay. You're saying, like like, what's reflected in
the IRR,
data? Or Yeah. Oh, okay.
I make no claims as far as the
accuracy. I'm not,
I,
I don't, you know, in the in the
work I do, I I rarely use that
other than, like, in this pursuit.
So,
I would
I would not be surprised by any kind
of data gaps that exist there. Interesting. So
I I didn't I don't think it's it's
meant for a, you know, like a,
Well, it seems to me it seems to
be responsive that way. Go ahead. Yeah. It
seems to me that part of the problem
we face here is that when the Internet
first started
oh my goodness.
Telling my age. Whatever. Anyway,
there was a high correlate. Things were fairly
steady.
Like, you could look
at an AS number or an AS path,
and you had a good guess as to
geographic regions. You had a pretty good guess
as to what prefixes should be there.
If you'd been on the on the d
f z long enough, you just kinda knew
these things.
Nowadays, like you said, Doug, with with DDoS
in particular,
and also even caching and Anycast,
and
all of the other insanity that we see
in routing flexibility,
the way people are using the routing system
to do a lot of things it was
never designed to do.
My sense is that there is just not
the correlations just don't exist the way they
used to.
There's there's way more
exceptions. Or they never existed to begin with,
and we are just fooling ourselves when we
collect these memories from old Internet.
Like, yeah. Yeah. We we used to know.
And maybe we never knew, but we you
know, the scale of our mistakes just did
a bubble up to nationwide news
all the time.
So,
you know, at a previous,
employer, I I phased out a lot of
IR based,
filters
because we concluded, like, hey. There is a
business risk
in loading these giant configurations
into the equipment that is already
overburdened
by the task that that the the equipment
has to fulfill in the normal line of
duty. So, like, adding megabytes of conflict to
that
in retrospect
are not achieving what we hope they would
achieve. They're not making things more secure. They're
making things unpredictable,
brittle,
and using a lot of resources while doing
that. And when you
frame it like that, like, hey. We're making
our network less secure with this routing security
initiative.
Yeah. It's it's a bit weird in the
beginning, but
eventually, people warmed up to the idea.
And especially
when,
I contrast it like, okay. If we remove
IRR, we can
supersede it or replace it with or substitute
it with some other technologies that have better
scaling properties.
Like RPKI,
ROV,
ASPIT. I I I wanna talk about something
you may not expect. It's a non RPKI
solution.
Because routing security is
more than just RPKI.
The one I wanna talk about is RC
nine two three four, BGP open policy.
And it's a really, really neat trick. So
it doesn't require a central database
that can go out of date or be
still or be insecure.
All it requires
is a per BGP announcement
marker.
It's called the only to customer
path attributes.
And the the routing equipment that recognizes this
new BGP path attribute
can use this to
understand is a path
possible or not.
Is there a route leak going on? Yes
or no?
So how this works is you and I
set up a session,
and on this session, we configure,
the relationship that we have to each other.
So I'll configure that I am the
provider, the vendor,
the upstream, like, whatever word you wanna use
for it.
And on your side, you configure that you're
the customer.
And then if both are,
routing devices support RFC nine two three four,
what happens is that this
bhp path attribute is added to the announcements,
and that helps restrict the scope of propagation,
in such a way that it's no longer
that it would not violate that model of,
valley free,
routing leaks.
So
long
story short, but if I configure you to
be a customer and you configure me to
be
the upstream provider, the routes you receive from
me
automatically
will not be propagated
to your other upstreams.
So no cryptography
involved, no central database.
It's per prefix.
So the information is packed
right where it's needed to assess,
the probability and possibility of apps.
And and it there's almost no memory consumption.
So it's it's a really, really cool technology,
that is almost so simple to use that
people might be skeptical, like, how can this
be useful?
And and and most implementations
support this, by the way. Just for people
who are wondering,
as far as I know,
most implementations support this. Is that correct, Dov?
Is that your experience?
It it is a relatively new technology.
So the RFC was published, whatever, two, three
years ago. Mhmm. And you'll see that open
source BHP stacks have support for this mechanism,
like OpenBHPD
or Berth.
And in the commercial off the shelf world,
the the hardware based vendors,
it is coming. So if you use, like,
this year's release or last year's software release,
chances are,
there is support for RC nine two three
four.
And if not, you gotta ask your vendor
like, hey.
I wanna simplify my operations and make them
more reliable and more trustworthy.
I want support for
nine two three four.
So so in the past, if you weren't
a transit provider and you're just a customer
multiple networks, you'd use you would do it
use an AS path list that's basically empty
to do this. Right? And you would use
that. Is this a replacement for that then?
Or is it more
It it's it's sort of a replacement, but
it's also something new. So instead of trying
to port
the existing functionality
of ASets in the IR and
blindly copy over that concept into the RPKI,
something entirely different happened. Like, a fresh look
was taken at what is actually the problem
we're trying to solve, and a very different
solution direction came out of that. So
it's not a one to one replacement of
technologies that we've used up until this point.
But it it does,
substitute,
or overlap with
the intentions of the technologies that we should
be deprecating.
I think the other thing is, Tom, is
that instead of you expecting your customer to
do the carrot dollar sign
configuration
on all of their routers,
this is actually simpler for them. They just
say, yes, I'm a customer. In fact,
as a provider in Upstream,
you can say, I know you're my customer.
I'm not accepting your advertisements.
I'm not accepting your beer your peering session
until I see this capability negotiated
and and stuff. So that so that actually
gives you a bit more control.
And it's incrementally deployable.
So,
about,
eighteen months ago, I I reported on the
North America network operator list, that a Internet
exchange to Internet exchange leak was prevented. Thanks
to
the OTC,
the the nine two three four attribute.
Because what had happened is that the Internet
exchange in Calgary, Canada,
uses this mechanism,
and,
France IX had also deployed this mechanism.
And then there was a a large global
carrier that was connected to both Internet exchanges.
And for some reason, it was picking up
routes,
from the Paris Internet exchange
and then propagating those to the Calgary Internet
exchange. So this would be
a a lateral leak. So it's a peer
to peer. It it violates the the valley
free,
conceptual model.
But because the Calgary
route server recognized
the attributes and it recognized that it came
from a,
a peering participant,
it could recognize that the path was implausible
and it would reject the routes and the
loop stopped.
So on the global Internet, at that point
in time, only
a handful of ASs had implemented this technology,
and it was already giving us benefits.
And I think that's really cool when
even with limited deployment,
a new technology
immediately gives benefits to the participants.
If you gotta wait for everybody to do
something,
then you're gonna be waiting for a long
time.
Especially on the global Internet. Yeah.
Right. Right. So don't get it. So it's
Is there any data there or anything that
you wanted to talk about as far as
how common these still are?
Are are people still using these
AS sets? Oh, the AS sets? Yeah.
Yes. I I I think they are still
in use for lack of a better,
mechanism, and everybody kinda has to come up
with some way to,
yeah, mitigate the the issues. Go ahead, Darren.
Better mechanisms,
but we have not yet reached global community
consensus
what those mechanisms
are. So it feels like there's
an issue of education and outreach like,
hey. Instead of using IRR based,
deployment strategies, you should be looking into
the OTC attributes, into RPKI original validation. Maybe
start a pilot with ASFA filtering or,
use maximum prefix limits or like, there's all
these
steps you can take to improve the security
posture of your network, but you have to
be aware
that
new approaches exist and that they are overtaking
the older approaches.
So it sounds it sounds to me like
we need
an an Internet scale
routing
routing control plane protocol. Right?
Because, like, we have all these things in
different chunks. We have these IRR things with
ASX. We have these features that are coming
along. Like, it I don't know. It almost
seems like we need a,
a a management plane
sort of solution that you can Another plane
on top of the plane. Yeah. Yeah. Or
You encode all this stuff in it. Replace
or replace BGP with something more rational.
Okay. There's a there's a rational thought, Russ.
That's another that's another podcast, I think.
I I guess I guess what gives me
hope like, I think we're we're we're we're
talking about problems this whole time just about,
here, you know, the OTT OTC discussion of
this, solutions of just, like, how, you know,
how how difficult this is. You've gotta
anything you're gonna try to waste onto the
entire Internet is,
is a really difficult challenge. But I would
say that,
the experience with getting RKIROB,
adopted to the state that it's in, I
feel like it gives us hope. And a
lot of credit goes to Job for all
of his evangelism,
on that, touring the world trying to get
convince people to to do this.
And, and I think it is it is
possible, but it takes a lot of effort
and a lot of time to get,
a lot of the Internet, a sufficient amount
of the Internet to,
adopt a new thing, but, I don't know.
I I feel like my role in this,
at least last year was like, alright. Let's,
let's just let's start our conversation around this,
as an issue because I feel like everybody
kinda knows this, who's who's knowledgeable,
but there's not, like, a a open conversation.
I feel like that was helpful. I didn't
have a ton of solutions. Job's got a
lot of ideas here about o t OTC
and then even ASPA. So where an AS
would assert
who are its transit providers,
in a in a RTI, you know, system.
Others could then use that to pick pick
out a value free violation of AS path,
reject that route. So that's those are those
are two approaches to deal with these adjacency
leaks.
The ASPA thing is is very early on,
and,
you know, we'll we'll see how it goes.
But,
I don't know. There's there's there's hope. It
just doesn't take, it's not gonna be, you
know, in a year that you you get
a year. Feel that. But, you know, the
listeners of this podcast, if you're doing a
greenfield deployment, you you're getting some new modern
equipment. And with modern, we mean anything produced
in the last few years,
and you have OTC support,
I would use that
anytime,
instead of,
recreating an IRR based workflow.
Because with OTC, the configurations become much smaller,
much more precise,
much safer
even though it's a smaller configuration. So maybe
some old timers will be like, hey. That's
counterintuitive.
We thought safety
comes from having a multi megabyte configuration, but
that's
that's that's not the case anymore.
And
using
the modern approach,
it's gonna
it's gonna give,
provide less risk to the business.
So if we, you know, popularize
this this knowledge and
and tell people, like, you know, don't don't
start new IRR projects. Like, really evaluate. Are
there other approaches I could be using?
Then the answer to that is yes.
In in 2025,
people should be hesitant to invest in IRR
based solutions.
And 2026.
Yeah. Sorry. Yeah. Yeah. Yeah.
Twenty one days into 2026,
and I still haven't fully settled in this
year.
All your checks have been the wrong day.
So yeah. Cool. Awesome.
Okay.
So
get the word out. Don't use IRR based
solutions. Use other solutions. There are solutions out
there. I think that's a good that's an
actually a very, very good,
place to leave this and to get people
to,
to do what they
need to do, to just, you know, get
things done and get security increased on the
global Internet.
So
anything else before we wrap up, Tom? Any
more questions you wanna ask?
No. No. It's been really fun. Thanks, guys.
Alright. And Doug, any final
comments? Any I think that covers it. I
think we did. We covered it well. Alright.
And, Jobe,
come to Amsterdam. Right, Jobe?
Come to Amsterdam and let's deprecate
IRR together.
Yeah. That would be great. Alright.
So, Tom, where can people reach you if
they want to?
LinkedIn.
The the length of the pause varies. Depends
on the mood that Tom is in. And
we're not really sure what the the correlation
is yet, but there's We need more data.
We need more data.
Alright.
Doug, where can people reach you if they
want to or get in touch with you
or follow your blog? So, yeah, you can
find me on, on LinkedIn.
I'm still on
x and Blue Sky and Mastodon. I'm in,
I put a lot of stuff in all
those channels. Do you blog over at Kentech?
I think you do. Yes. I do. Thank
you for mentioning. I I, my company would
be disappointed if I didn't mention it. But,
yeah, so I wrote it. I, I write
quite a bit for, for the Kentic,
blog about,
both
this this kind of topic of kinda, like,
the routing,
kinda gearhead, I call it,
technical, parts of the Internet as well as,
a lot of stuff that goes on, at
the intersection between geopolitics and and Internet technology.
So Cool. Might find that interesting. Awesome. And,
Joe, where can people follow you if they
want to?
Just go to my website,
b s d dot n l,
and,
send me an email, joe@bsd.nl,
if you wanna talk shop about
strategies
to improve security and reduce business risk.
I,
I really like tinkering with large
networks and finding solutions that work well for,
for everyone. And, you know, if that means
that the entire world needs to change a
little bit, then,
hell yeah. Let's do it.
Not afraid of the world. That's Job.
Alright. Cool. Well, I'm Russ White. You can
find me here at the hedge at rule
eleven dot tech,
on LinkedIn, sometimes on x, Who knows where?
I'm really easy to find if you wanna
find me, so just go look around.
We know we live in an attention driven
economy
and that your attention,
is important in the world, and we thank
you for listening all the way to the
bitter end of this episode of The Hedge.
Turn off your IRR based stuff,
and we will catch you next time. And
turn on OTC. Like, don't go flying without
any precautions.