Join us as we gather around the hedge,
where we dig into technology,
business, and culture with the finest minds in
computer networking.
Good morning, Tom.
Good morning, Russ.
There are still holes in your bookshelf.
Yep.
Yep. And There
it's a sign, Russ. You need to write,
like, you need to write, like, five more
books.
In my spare time. That's what those holes
are for. Yeah. Yeah. In my spare time.
Yeah. I don't know. I I probably do
need to write more books, but I just
need to write more. But this has been
a busy this has been a really busy
six months. I don't know.
I I for those who don't know, I
teach at University of Colorado once a year,
and I'm in the middle of University of
Colorado. And I'm also in the middle of
that course.
And I'm also doing a bunch of recording
for a training thing that I'm doing, and
I'm trying to develop new material for Pearson
and, you know, just and, and, and, and,
and. So springs are always
spring is always, for some reason, hyper busy.
It it's, you know, it's like I wanna
shift some stuff to fall, but so far
I've figured out how to. So okay.
Well, you can okay. You have my permission
to wait to write those five books until
the fall.
Okay. The holes in your bookshelf will wait
till fall. I see how that's gonna work.
Okay. Great.
And today, we are joined by Dell Darryl
Swear? Swear? How do you I'm sorry. Swear.
That's right. Yeah. Swear. Darryl Swear. Yeah. Alright.
And where are you, Daryl, physically? Where do
you where do you hail from?
Yeah. So I hail from Shillong. It's a
small city, small town up in Northeastern India,
and that's where I currently am physically located.
So,
yeah. That's where I'm from.
Okay. Cool. So close to home. I'm actually
not that far from home. I grew up
in the Southeast, and I've never moved very
far out of it.
The furthest I've actually lived from from my
home, from my original from, like, where I
grew up is actually New Jersey.
Isn't that crazy?
Like, I've stayed on the East Coast my
entire life. Somehow,
I've been in technology
for thirty years, and I've never moved to
California.
I don't know if that's a you should
feel sorry for me or if that's
a
I'm not sure what I'm not sure which
that should be.
So let's talk about out of band networks.
Right?
AWV. So
out of band networks are a thing. And
I know that, for instance, when I was
at LinkedIn, we had an out of band
management network.
And I also know that we got rid
of it
to some degree.
We we had
it, but we had it we we moved
it from being its own separate infrastructure, physical
infrastructure,
to being,
a verf.
So it still had its own IP addressing,
but it was on the same physical infrastructure.
But I don't know of anybody else that
I've worked on their network.
I can't think of another single network I've
ever worked on that actually
that that I have managed or been in
the part of management. I've seen them in
other networks
where there's actually an out of band network.
Isn't that crazy?
Like, you would think it's the most natural
thing in the world to have an out
of band network. Why why not?
So
yeah. You know? So so, Darryl, what's your
take on this? Like, where where do you
start thinking about it? In in my opinion
and in my in my experience, right, you
I agree with your observation that most,
networks out there, regardless if there's service provider
networks or data center networks or enterprise or
something in between,
most of them,
for reasons I cannot fathom, they do not
have an out of band network, let alone
let alone an in band management
framework of some sort. A lot of these,
networks out there, they they don't have
a
streamline,
management implementation,
right, let alone an auto band network.
And I think that's a bad position to
be in because when
disasters happen, when things break,
whether that's, you know, whether that's a layer
two plus or whether that's a physical disaster,
layer one and below, including potentially layer zero.
You definitely should have an outbound in place
in some shape or form,
to ensure that you can restore connectivity and
access to your relevant systems and devices and
so on and so forth. Right? Like, in
the past, five, six years, we've seen quite,
a few number of,
large scale outages. Right? Like, there was the
famous,
Facebook slash meta outage back in 2020, or
was it '21? I don't remember.
There was the Optus out outage in Australia,
and there there was various other outages that
I I can't recall right now.
And from what we can tell, based on
the public information on some of these
instance, these outages,
it wouldn't be very difficult for somebody to
gauge that if these,
instances
had a properly architected out of band network,
those disasters could have been mitigated
pretty much a lot faster than they were
because, they didn't have a sufficient
or properly architected out of band network. So,
yeah, it's a mystery to me as well,
Tom. I have no idea why most people,
don't take out of band like a,
natural first step to building a network. For
me personally,
for the greenfield networks that I've worked on,
my first step is building the out of
band network. I I don't build the edge
or the core or the aggregation and access.
No. First step's always gonna be the out
of band network because then that's how I
myself, along with the staff or people who
are in charge of the network will actually
log in and make changes to the rest
of the network through the auto band.
So that's how I personally see it, but,
I do I do wish and hope more
people would adopt an auto band approach at
least some extent, for sure. There's actually a
funny story from Cisco Tech about this. We
used to have,
ship kits,
or mail kits. And what it was was
it was basically a hard case
with a modem
and all the correct cables.
And if you lost access to your router
and you called TAC,
we would ask you for an address, and
we would overnight FedEx
a ship kit to it and ask if
there was anybody physically on-site
who could just, you know, find the right
cables that would fit
and put it all in there. And it
was like a modem and everything else and
just please connect the telephone line to it.
And later, they became five g or four
g, not five g, but four g. Right?
Later, they became it was a four g
cell phone
with the right cable connectors and a much
smaller box because there was no modem. And,
like, you would just connect everything together,
and then we could, from the tech, get
in
and go get you access back to your
router. But this see, that that seems ridiculous
to me, because you're at the point in
your network where you have to ship a
physical box
to get back in. That's that's just crazy.
I I think a lot of times,
like human nature works against us here.
You build you build networks, and especially if
it's a if it's a a big network
that generates revenue,
the the network itself has to be resilient.
And so I think a lot of times,
it's just really it's it's easy to overlook
out of band because, well, I've I've built
the network to be resilient. If I have
a failure here, I've built it to route
around this problem. And so we we lean
on the resiliency of the network to,
kind of, you know, where management management traffic
goes up the wrong direction or around the
horn or whatever,
we just we just it's like, well, we
invested in this resiliency, so,
it's not a pressing issue right now. And
I think for for most I would say
for most operations, there's, like, a whole bunch
of things that are fires that are burning
hotter than that. And then, unfortunately,
what happens is,
eventually an outage comes along that you didn't
anticipate in the in the design of your
network, And, the resiliency that you built didn't
cover didn't cover this case, and so you
see some of the outages like the ones
you were talking about, Daryl. But, I think
it's just it's just hard to invest in
something. It's it's like paying insurance.
You know, I think there are a lot
of people, at least in The United States,
that wouldn't pay any insurance at all if
they weren't forced to by law.
Because we don't wanna we don't wanna pay
for things that we don't that aren't hurting
us right now.
So it's kind of a human nature thing.
And it's not just cost. It's management. Right?
This entire out of band network needs to
be automated,
configured. You need IP address space.
You need tools.
You need to monitor it and make sure
it's working.
Because a lot of times, it's easy to
put an out of band network in place
and then never use it. And then when
you get to the point where you need
it, oh, that doesn't work. Oh, oh, that's
a problem.
Right? Because you never use it. So I
don't know. I mean, there's
there's lots of stuff around this area.
So is there any other justification?
I I have justifications, but I'll I'll ask
if you all think of any.
Is there any other justification besides the, oh,
my network is down. I can't get to
this site or this host or this router
for an out of band network?
Like, what else if I was going to
my boss and saying, we should spend money
on an out of band network.
What what what should I say? How should
I like,
beyond, oh, there's a catastrophe
waiting in the wings. I have an idea
too, but I wanna hear from Daryl. Yeah.
Yeah.
So
I use it now. So there was this,
there was a client of mine,
that wanted to,
this client of mine, he, he, he was
a business owner, right? He was a businessman,
but he was not a, an engineer by
profession. Right. So he wanted me to explain
to him,
the importance
of an out of band network beyond just,
you know, disaster recovery, like like you just
said.
And I explained it to him this way,
like,
the production network would
be similar to a public highway,
whereas the outbound network would be similar to
maintenance tunnels
behind or on the side of the highway.
Or, for example, if you have,
public water systems, public sewage systems, there's bound
to be maintenance tunnels or maintenance systems in
place, right, for
those public,
infrastructure and services.
And that is how I try explaining it
to to this client of mine
couple couple years ago.
And he liked the, the analogy, the the
the ID concept behind it. And it's not
just for disaster recovery. When when I build
an outbound network, I
encourage
the
the the humans behind the network to
exclusively
access their remote devices
and systems and software applications, etcetera,
exclusively, again, truly out of band networks. So,
essentially,
the only point of ingress into the management
plane of the organization
is exclusively through the out of band network
for 99% of the time. Now now there
may
be edge cases. There's always gonna be some
edge cases or some edge corners here and
there where this approach will not work. There's
always gonna be that 1% exception.
But for 99
of the time when, when I built the
out of band networks,
for these very very use cases,
I always encourage that it's supposed to be
used by employees all the time. So if
you
if you go for a VPN model, the
VPN will be behind the outbound network.
If you go for the,
the, what's it called again? A SOCKS proxy
model, there there are some company that don't
do VPNs. They prefer SOCKS
proxying and
zero trust and all those fancy layer seven
based login and security measures, etcetera.
All of that goes through the outbound network.
It the the only ingress point for authorized
staff, employees, and management into the network infrastructure
or any software applications running behind it, for
example,
NetBox,
LibreNMS, you know, some of these applications. The
only way for a human to access those
applications
would be through the out of band network
because it also helps with network segmentation, you
know, keeping the production revenue network
in a different segment, keeping the management network
slash out of band in a separate segment.
It also helps with the security posture,
in in my personal opinion. I don't think
it's a good idea to mix
your
management traffic
with your customer traffic. I don't see a
good reason to be doing that. And since
if you have an out of band network
in place
and you have your network devices in the
production network, for example, let's say you have
a router in production network and,
it has a management Ethernet port, not to
be conflated with the console port, but a
management Ethernet port.
So what we can do from a security
standpoint is to ensure that your management related
applications,
such as API access, SSH,
streaming telemetry, etcetera, all of that should be
configured
to hard bind onto the management interface
slash management v r f to ensure that
only the management network slash OOB can access
those services.
That way, ensuring that nobody on the transit
path or the public facing,
routing table of customers, etcetera,
will ever be able to talk through the
daemons listening
behind that management of VRF. So it it
really I think it really helps with security
and segmentation for for everybody to have some
form of outbound network, not just to act
to use it when there's a disaster, but
to proactively
use it to manage, automate, and orchestrate the
network.
So so
you're actually on board with not a physical
network, but rather a VPN or proxy based
network as well as an out of band.
Right?
That's that what do you prefer do you
think that the physical infrastructure out of band
is actually better? I I I would prefer
the physical. A mix of both, you know,
for purposes. So you,
here's how I would put it. If I
would do priority one with the physical
out of band infrastructure and then for failover
or backup or purposes, you know, we we
can never not have too much You can
always just keep having Right? So you could
have some kind of cloud based virtual VPN
slash slash proxying
approach to it, but I would still prefer
for for us to have a physical infrastructure
for OOB,
as the first step. And then we can,
you know, have virtual
implementations to it. Even if it's a cloud
connected, I would say. Like, even if it's
via a cloud but is a separate physical
infrastructure,
I actually prefer the separate infrastructure myself as
well. Tom, any thoughts on that particular question?
Yeah. Yeah. I think you you want it
to run independent of the of the,
the the revenue plane, if you will, the
data plane that carries the operations of your
business.
Yeah. I think that's
running them separate is I think I would
assume that's pretty well settled.
You know, if you you don't really you
don't have an out of band. It's not
out of band if it's not separate, right?
It's in band if it's separate.
And so that's kind of to me, that
it's the definition of out of band is
separate physical,
physical transmission medium.
Interesting. So now if I were going to
my manager to justify, you can you can
critique my thoughts on this if you want
to.
It is this, that
just like in routers and a lot of
people don't know this.
But when you build a router, when you
are designing a router, boards,
chipsets, everything else,
In commercial grade gear and, by the way,
this is something that's different about Linux based
routers
than about,
like, Juniper, Cisco,
whatever.
There is actually a separate
often, there is actually a separate internal channel
for telemetry.
Because
it's it's an advertising it's a marketing thing.
You can't say, oh, my router can support
x gig per second
if I'm producing
x meg per second of telemetry
and I'm sucking it off my primary
backplane or network interface interconnect.
Now this is more true. By the way,
this is not necessarily true of
single box, single FEs. But when you get
into chassis boxes in particular,
almost every chassis box I've ever worked on
has a set of runs on the backplane
that are nothing but telemetry.
They're they're they're not there. They don't run
customer traffic, or they don't they don't run
through traffic. They only run internal into the
inside the box traffic.
Now if you think about it in those
terms, you realize
that if you are running
your
telemetry
over the same network as you're carrying your
data plane,
you're doing exactly the same thing.
You are consuming
bandwidth.
And maybe your bandwidth on your links is
very, very low, And you're like, I don't
really care. I only run 3%.
I only run 5%.
But
there's two problems with that. The first is,
there's always the issue of
measuring
things, changing things,
which is something we don't think about very
much in networking.
But it's true.
When you measure things, you change them.
And the more you can separate the measurement
from what you're measuring,
the less you're going to change things.
So this is just a,
this is just the way that things work
in the real world.
And the other thing is, again, when you
come down to an emergency
or you have massive problems, like, if I
have a routing loop
that's consuming 80% of my link,
it's
stopping
my telemetry
traffic from making it to my system
that discovers
the routing loop.
Okay?
Like, stop that.
Don't do that.
So to me, when I build the out
of van network, whether it's proxy or VPN
or whether it's physical infrastructure,
I'm gonna put all of my telemetry on
that network.
Not just remote. I'm gonna put everything I
can on that network that is
what you know, whatever it is. So I'm
gonna put my network management system and my
automation system
on that secondary network
and never run
it. I'm gonna have a clear, clean separation.
So I don't know.
Maybe maybe I'm a little bit sensitive to
the measurement changes things, and I've seen cases
where
I was working with a bank,
and they were getting eEdge RP SIAs, which
tells you how long ago this was.
And
we put
packet,
we put packet dumps on all the interfaces
to try to figure out what was going
on because we couldn't figure out from the
EIGRP perspective, like, why we were getting stuck
in actives.
And it turned out
that every time we did a packet dump,
because we saw an SAA,
all the traffic in the interface queue was
management traffic,
which
made it impossible to troubleshoot the actual problem.
Right?
So sometimes you just gotta, like,
stop,
Take this stuff out. Make it separate
because it makes it easier to troubleshoot. It
makes it easier to see what's going on.
So, Adam, thoughts on that at all? Daryl,
do you have start with you.
I I agree with that viewpoint,
Ross. I I think,
in general, when we deal with technical systems,
complex systems, as a matter of fact, right?
These are very complex systems in today's today's,
day and age.
I think separation
as much as you can
is probably a good thing. You know,
there there was,
I I I there was a paper back
in college, that we had. It was a
software engineering related paper, and there was a
line that stuck with me. It it wrote
something like like this. It said, the the
best approach
to writing code and and software engineering would
be to have
high cohesion and low coupling.
So I would really prefer for the network
to have the same kind of principle where
the coupling
is minimized as much as you can, and
then you can increase the cohe cohesiveness
as much as you can and as much
as you want. But I think coupling of
these components,
whether there's management or or control plane or
routing protocols, etcetera, etcetera, I don't think that's
a good idea because of exactly the kind
of examples you shared with us. I think
low coupling and high cohesion should be applied
to network engineering in general, not just in
software engineering. I think we need to apply
that to, networks as well. So I completely
agree with you, Ross, on that. Yeah.
Yeah. And and I think the first counter
to that would be, is there any historical
data
about
and and there may not be any historical
cases in a particular case where they've been
locked out of a router and had to
ship somebody on-site.
And and
but if there are,
how much did that cost?
And and what what does the cost differential?
If I lose a store
for a week
and I cannot do any access to it,
what am is that store gonna stay open
if my margins are that tight?
Right?
So there's always like this so I worked
on one network once, which was a retail
network, where they solve this problem a little
bit, like, perhaps oddly,
is they actually had three different connections into
every store, physically.
They had some type of a local provider,
whatever it was, at the time, you know,
d s three, whatever it was, it happened
to be.
Then
they found a local cable
provider
that was completely diverse, and they didn't buy
a business level plan. They just bought, like,
your bog standard
$20 a month, send me your modem,
give me an Ethernet connection,
and I'm gonna stick it on the back
of my store router.
And then they had a satellite connection.
All three. Now, perhaps, that's overkill.
But what they found was was that to
justify it, the way they did it was
they said,
we are going to have something in the
router
that if the store loses connectivity,
we're gonna ship all of the credit card
information over the
cable modem
so we can continue doing business.
We may not have inventory.
We may not have video. We may not
have video cameras. We may not have anything
else, but we're going to process credit cards.
And we're going to have two ways of
processing credit cards. So this store will stay
open
all the time. There will not be a
day when this store is closed unless there's,
like, a hurricane,
you know, and it takes out all connectivity
and power.
But if the primary provider goes down, well,
you know, we'll lose video feed from the
store.
Oh, well. But we won't lose credit card
transactions.
And so that's the way they justified it,
was they said,
this increases your revenue over time because you
can look back at how many times that
primary link has gone down during the year.
And even if it's for two or three
hours that they can't process credit cards,
then, you know, you're in a completely different
situation
when you can justify the cost of that
second link just on normal operational procedures.
And then that can
become your
Yeah. Right.
Right. Yeah. ATM machines are the same way.
A lot of ATM machines are double connected.
Not not all of them. That used to
be more common, I believe. Maybe it's not
as much anymore.
But there always used to be a separate
connection from the ATM machine to the central
bank
office
and the connection through the local office.
And that was for this very same reason.
The bank loses revenue when the ATM machine
doesn't work.
And a customer comes by and the ATM
machine doesn't work,
they're gonna remember that in a month, and
they're gonna go find a different ATM machine.
Right? And so it's it's not just revenue,
it's mindshare.
Right? And and so we we we do,
Tom, like you said, we do need to
think in these terms,
creative ways to create an out of band
network and justify it.
So that's that's kind of that's a that's
a thing with me. So, Daryl, I don't
know. You haven't said anything in a while.
Tom and I have been, like
No. You did mention something interesting earlier. So,
you said you you mentioned there was a
business that actually made use of satellite for
the,
you know, OOB access. Right? So interestingly, I've
done the same thing except with Starlink. So
there was this,
wireless
ISP project I did. I believe they were
based in Montana up in the mountains.
And so as you can probably imagine, there's
not a lot of fiber connectivity in in
that particular
area slash region that they were operating in.
So
I
got the idea that, hey. Why don't I
just use since they already had Starlink connectivity,
why don't we just put a Starlink up
in in this pop, in this,
in this site and use I p v
six from Starlink
for the OB access. So that was it.
Like, you know, I could any of us
from anywhere on a planet could then remotely
access those
devices on that site up in, up in
the mountains
over I p v six, over Starling without
having to worry about,
you know, CGNET and all all those kind
of stuff, you know, like a reverse proxy
and all that kind of stuff.
So,
this is also another key point that,
for any OOB project that I do, IP
v six is a must have because, like,
I would like everything to be routed. I'm
not a fan of NAT. I'm not a
fan of all these layers of, you know,
proxying and and NAT ing and triple net.
I've even seen triple net before. I'm sure
some of us have seen triple net. I
I I'm I'm I'm just wondering connection is
triple netted.
I'll bet I'll just about bet everybody's home
connection is triple netted. So sorry. Go ahead.
Yeah. Yeah.
So, yeah, I I I think IP v
six is also very important for the OOB
because it's also for simplicity. Right? If everything
is just routed, like, you don't have to
worry about all kinds of these problems with
remote access. If everything has a goa on
it, of course, you're gonna have a stateful
firewall protecting those devices behind,
the OOB network. You know, from from a
security standpoint, you can have s, SPI
in front, in in in between those devices
and the edge of the OOB network, protecting
them from, from the public Internet with a
public IP v six address. But I think
that was that's that's an important,
attribute to think about. And as far as
creativity goes with OOB, I think Starlink
I think we'll probably see more people making
good use of Starlink for business applications, including
out of band network.
I think
styling is a very interesting,
tool in the toolbox for for network engineers
going forward is what I think.
And now the other interesting bit that,
Tom mentioned was,
you know, a lot of network engineers are
waiting on the finance
guys or finance team of their employers or
company,
to try and address some of these, concerns
or issues we have.
And,
it it does kind of, like, debate from
the original topic here, which is out of
band networks. But I I
I think in in this day and age,
it's very difficult to just be to just
quote, unquote, just be an engineer
without learning some amount of finance and accounting
and some of the business side of things.
Right? I think
I think we we we as as humans,
we as a civilization, I think we have
grown very complex,
and we have all kinds of,
of facets to our very fabric of the
society whereby you can't just
be a quote, unquote,
network engineer. I think you need to learn
quite a bit more than that to actually
be able to survive in that given role,
and this is not just for network guys.
You know, this applies applies to any kind
of job out there in the software,
or whatever it may be, I think.
But but it also does create some layer
eight challenges as I like to call it
because
in many of these cases, and this is
from my own conversations with a lot of
our fellow professionals in the industry,
the ground reality is oftentimes
network network engineers or network staff are looked
upon as a call center by the stakeholders
of a business.
And what that means, the bottom line is
a lot of these engineers are underpaid. They're
they're not really paid what their expertise brings
in or what value they deliver to the
company. They're underpaid, so which then leads into
this whole cyclic,
dependency problem whereby
they're not paid enough. They know what the
problem is. They know what the solution is.
But, again, they're paid enough so they don't
bring these these concerns to the leadership
or finance department of the company. So then
the problem never gets solved. They leave the
company. Someone else comes in. Same problem. They
leave. They go. There's a lot of churn
in that ways, and nothing gets resolved at,
for ten, fifteen years down the road.
So I I I don't know what would
be the right,
way to tackle this issue, but,
but it's definitely an issue out there. It's
a little it's a layer e issue. It's
less of a engineering problem and more of
a human issue, I think. You know, a
layer eight issue, maybe even nine and ten
in some cases.
So, yeah, maybe, you guys have some thoughts
on that. So I completely agree. We've really
bought too much the idea of
jack of all trades, master of none. I
hate that saying. I think that people
really overplay
it.
And, you know, oh, you think you know
about philosophy and you think you know about
networking. You think, oh, you can't you can't
know all that stuff.
No. That's wrong.
Like, you as a human should be constantly
expanding the scope of your knowledge
and and the mental maps you have of
the world
in useful and helpful ways.
And that's just and, of course,
the younger you are, and this is not
to say anything bad against young people or
anything, It's just a life experience thing.
It's
you know more after you've been doing something
for twenty years and dealing with a lot
of different businesses
than you do when you've only been dealing
with it for five years. And you might
think, oh, five years is enough. I know
everything. No. Trust me.
You'll learn more. You should be learning more.
It should be your goal to learn more.
And so that is that is something that
I don't think we take seriously enough
in
our world,
in technology in particular.
We tend to idolize people who know
one or two things.
Like, we we we really idolize the guy
who's a c coder, who knows every in
and out of and can tell you, you
know, oh my goodness. That's c 19 80
four, and that's c nineteen eighty six, and
that was put into the spec on this
date and by so and so for we
really idolize that.
That's really that's the those are really cool
people to be around, but the honest truth
is they're not as useful
to the real world business
as somebody who
knows a broader range of things. So I
think that's that's always true. And going back
to out of band networks,
I think knowing a lot of things helps
you justify building a network that's better for
the business.
I think that's that's that's an important
it's not just for me, it is for
me, it's not just for me, it's also
for the business.
So I think
those are very,
important things to think about.
So that's kinda my take on the whole
range thing. It's been my take for a
long time. There's a great book called Range.
I I love that book. It's it's being
being a fox in a hedgehog world, I
think is the subtitle or something like that.
Or the or the value of something like
that. Anyway,
the
value of the generalist in a specialist world,
something like that. So it's very I think
it's a very important thing to to get
into and understand.
So back to out of band, how this
all relates to that is, again, it helps
you justify things.
And it helps you build a better network.
Starlink, I think, is interesting.
Now, there is a new attack surface when
you do this. Don't think that there's no
tradeoffs.
Right? This is something we often do. This
is all beautiful. It's called build build out
of band networks.
You have to secure the out of band
network.
Like, it it's got
it it needs SSH or it needs something.
Whatever you normally do, it needs something.
Don't don't assume
that no one's gonna find the right IP
address.
Yeah. That's that's not gonna happen. So Well,
and we're and we're and we're tempted, sometimes,
building out of band networks to say, well,
if we're gonna use this in an emergency,
it needs to be really easy. It needs
to be, like, I don't I shouldn't have
to go through any any special hoops. I
should because I'm gonna need to get in
there fast. And you have to you have
to fight that. You have to just it's
it's another network that has to be secured
like everything else. Yeah. Yeah. It's a part
of your attack surface, and you should you
should evaluate
it in that way. So And and it
quickly
expands once you say, oh, this is a
great idea. I'm gonna start doing it. All
of a sudden, you'll be like, oh,
how do I get how do I get
my authentication systems under this segment? How do
I do I have to put two interfaces
in everything?
Do I you know, it it starts it
like, it's not a it's not just a
matter of,
throw in a, you know, a cheap cable
modem
and put a firewall on it, and we're
good to go. You have to you have
to think through it. It requires it requires
design work. Yeah. You know,
speaking about security for out of band networks,
I I I think I like, the way
the way I take network security,
POV point of view, the way I, you
know, the way I tackle network security, right,
it ties back to my
background where I started learning networking with with
Linux based systems. That means Linux and the
filter framework. So I'm pretty familiar with Linux
Netfilter Framework.
And so when I built these the the
security configuration
behind these OOB networks,
I
combined the Linux Netfilter Framework Packet Flow
knowledge
with the ideology
of defense in-depth, you know, the military strategy
for defense in-depth. Right? So I I combine
these two approaches
into designing the net sec aspect
of the OOB network. Now,
of course, realistically
realistically speaking,
most most of the businesses out there, they
they don't have the required
in house expertise.
And what I mean by that is normally
a software engineer,
to build a CICD pipeline to automate these
OOB networks and the production network as part
of it.
So in a lot of these cases, these
these these networks are are still manual driven.
I mean, as much as some of many
of us would hate to admit it, but
a lot of networks are still manually managed.
Automation is not the norm. It's the exception,
unfortunately.
It's still very expensive,
to hire a expert who can build a
CICD pipeline,
for your automation. So,
when I build these security
configuration parameters,
I make good use of the prerouting chain
on on these Linux based device network devices.
So in the out of band network, I'm
I I some might say I'm biased, but
whatever.
I'm biased to use Linux centric network devices
or operating systems in the OOB network. Why?
Because I can take a lot of advantage
with the prerouting
chain. So if if if you recall in
the in in the last twenty plus years,
there have been
very advanced
low level,
network attacks that takes advantage of how the
pack, the packet processing engine works on these
various operating systems. Right? Like, they could construct
a malformed packet with a certain
fragment in a certain way with a certain
byte size and do some real damage.
But if you drop unauthorized packets in the
pre routing chain, it doesn't even get very
far. Right? It gets dropped I mean, it
does pass the socket buffer, SK buffer of
the Linux framework,
but it doesn't get very far because you're
dropping it in the pre routing chain, so
it doesn't even enter the connection tracking table
whatsoever. It doesn't go there. So if there's
any kind of attack that is trying to
do something
post the pre routing chain, it's not gonna
happen because my prerouting,
prerouting
filter rules or ACLs
are dropping those package the moment they they
hit the NIC or the moment they they
process by the system.
So I, of course, like I had mentioned
earlier, segmentation is important and decoupling
topology in the outbound network that is kind
of similar to
a collapsed core topology, sort of similar to
that. So you have the
OOB edge router, which has your upstream connectivity,
whether that's a dedicated transit, that that's your
Starling, LTE, whatever you want. That's where the
OOB Edge is. And below that is my
OOB layer three distribution router as I call
it. Some people might call it something else.
The name is not really important.
And and on the on the layer view
distributions where I have the stateful,
firewall rules to protect my OOB client devices.
So that includes,
systems, physical server boxes maybe,
your other network devices, or and even your
PDUs. So your your your power your electrical
components that have Ethernet management,
those are also protected with the OOB, and
you can access those through the OOB.
So it could it's not just the network.
It's more than the network. You can have
IoT devices, you know, like door controls, that
kind of stuff, building control, access control. All
all that kind of stuff should be tied
down to the OOB network, not to the
production revenue network.
And, yes, security wise, they are protected by
a stateful firewall on the late fee distribution.
And the OOB devices themselves,
since they have rules in the prerouting chain,
they are themselves protected on ingress from any
kind of potential malformed packet that may be
trying to take advantage of
undisclosed, undiscovered security vulnerabilities
in the Linux kernel.
So that's how I tackle it. Of course,
there's a lot more involved once you go
to layer seven. Right? You have zero trust
and all these things, and it gets really
complicated. But but the principle is that,
I I really am biased with a Linux
centric approach,
but maybe there could be better approach that
I'm not aware of. But so far, for
me, Linux centric approach for OOBE
has paid off. It has worked pretty well.
It's actually probably less expensive too because you
can actually run Linux on fairly small boxes.
I mean, even down to some larger Raspberry
Pis, if you were that crazy about doing
it, that would give you a very inexpensive
way, and then your main con your main
problem becomes connectivity
into the back end
rather than,
you know, main managing and maintaining
large expensive boxes with spinning hard drives and
stuff like that. Yeah.
So I don't know. I feel like we've
kind of beat this to death. Is there
anything else you wanna talk about, Daryl, before
we wrap up?
No. I think we covered the the main
principles, I think, of the, out of band,
you know,
business wise and technical wise, I think we
covered most of the concepts. So, you know,
may maybe Tom has some something to say.
Yeah?
Tom always has something to say. He says
LinkedIn.
That's
right. But I'm saving I'm saving that for
the right moment.
Oh, boy.
Alright. Well, Daryl, where can people find you?
Do you blog or anything else? I I
saw your blog post on this, so you
must have an outlet. So
do you blog on a regular basis,
or do you Twitter? Yes. Whatever?
I do publish blog posts now and then.
I do have my own website.
It goes by my name, deaddelsware.com.
You can find me there. And on my
website, you can find links to other social
platforms, LinkedIn, Twitter, etcetera, etcetera. Yeah. Okay. Awesome.
And go ahead, Tom. Just say it.
LinkedIn.
I've been practicing.
I I don't think you're fast enough. K.
I'll work on it.
Alright. I'm Russ White. You can always find
me here at the hedge, rule eleven dot
tech. I log into x occasionally
and LinkedIn occasionally, so you might find me
there,
you know, if you wanna DM me or
PM me in one of those places. If
you have any ideas
for things you'd like to hear or talk
about hear us talk about on the hedge,
please let me know.
And, Daryl, thanks for coming. By the way,
you know, anytime you have another topic or
something you wanna talk about, please just let
us know. We're pretty open to new ideas
and thoughts. And I don't know if you
know this, but podcasts like eat material.
That's it's it's always,
we can always use more topics.
Yeah. They do.
So
alright.
Thank we'd like to thank you for listening.
We know that we live in an attention
driven economy, so your attention's probably the most
economically valuable thing out there right now. So
thanks for spending the time with us,
and
we will catch you next time.
Time.