Join us as we gather around the hedge,
where we dig into technology,
business, and culture with the finest minds in
computer networking.
Well, hello, Anil. Where are you physically?
Hi, Russ. Thanks for having me. I am
in San Francisco, California.
San Francisco.
Oh, cool. I used to work for LinkedIn.
So I know San Francisco well because their
headquarters is down on 222 Second.
And I used to walk from,
the Hilton
on
right up by Eighth Street. I don't remember
the cross street there, but and then I
walked down Eighth or Seventh or Fourth. Maybe
it's Fourth or something I'd hit.
And I'd run down Market
to get to work in the morning when
I was staying in the Hilton. And I
would stop at the Ghirardelli for for hot
chocolate because that's like
good hot chocolate.
That's great. And then before I left, I
always picked up something from the moleskin store
because my wife loves notebooks,
loves notebooks and pens. So I was always
going to the moleskin store going, okay. What
pen have a dye bought her from the
moleskin store?
Because it's right there on Market.
Yeah.
So,
I guess, maybe start here, and you'll tell
me a little bit about yourself. Right now,
we don't have any we don't have Tom.
We don't have Yvonne. So it's just Anil
and I for our regular listeners. You know,
you can be disappointed and hang up now
if you want to and stop listening because
we don't have Any of the interesting people
on. But yeah. So talk to me a
little bit about yourself, Anil. Like, what's your
background and kind of what you're working on?
Sure.
My background is I studied network engineering,
built networking systems for about
a decade at a previous company, and then
I helped start Meter about
nine, ten years ago now.
Meter is a networking company.
We build,
deploy, and maintain
networking
hardware and software.
Okay.
We design
everything from the hardware on up. So this
is everything for
routing, switching, wireless, security.
I know, Russ, you can see our rack
behind us. Yeah. Yeah. But it's it's our
rack where it's our,
PDUs,
firewalls,
switches,
indoor Wi Fi, outdoor Wi Fi, cellular,
where we design and build the hardware, like
the firmware, operating systems, distributed systems, APIs,
applications. So basically a full en route switch
company that does hardware all the way up.
And I Hardware. Yes. And and we work
closely with our customers and partners to help
deploy and maintain it as well. Okay.
And what we do differently sort of three
things in all these years of networking is,
you know, Russ, majority of the networking stacks
have been built through acquisition.
Mhmm.
We sort of decided that there should be
a company. Ironically, we're the first people to
do the entire rack together with a single
operating system,
single firmware, a single API for the entire
rack,
and fully integrated products across ISP management, routing,
switching, wireless,
SDN,
DNS security,
VPN,
IDS, IPS, all in a single platform.
Then we're vertically integrated in that. We also
help deploy these networks everywhere in the world
with our customers and partners. We work with
the myriad of channel partners.
And then our pricing model is a bit
different. We take on all the risk for
the hardware.
So there's there's no capital costs.
Okay. And when there's existing hardware in place,
we'll buy that out. And then when we
build new hardware,
we'll upgrade our customers for free as well.
So
so this is a networking company from the
ground up. Cool. So this is all very
operationally oriented. That's what I find interesting about
it, is that you're not focused
so much on the route,
routing features, speeds and feeds,
stuff like that. I'm sure you have plenty
of features, but that's not really your primary
focus. Your primary focus is on operations,
which takes us back to a conversation we
had with Mike Bouchong,
on the hedge
not too long ago,
maybe a couple of months. I always forget
when these things post, so I don't I
don't remember how long it's been since I've
talked to somebody about something.
That
operations is kind of the future or is
the future of network engineering.
We were round and about about that a
couple of times with Mike, and,
I pushed back on him on some things.
And he pushed back on me, which is
all good because
Mike and I are good friends, so that's
good. So,
I guess if you're so operations focused, it's
interesting the tack you're taking or the path
you're taking is unified operating system.
Now are you writing your own stacks, or
are you using open source like FR routing
or BER or something like that?
Everything is us from the ground up. And
okay, maybe to correct you just a slight
bit. We do care about the speeds, we
do care about the speeds,
we do care about every inch of the
hardware, we
are care designing the hardware. Our incentives are
very different because if you look at all
these legacy companies, they are building hardware just
so they can build it for as cheaply
as possible
and sell it for as expensive as possible.
Our incentives are the hardware has to be
great operationally like you're saying, because we're deploying
it and it has to work really well
because we only get paid if the service
is great.
So everything is in that light where we're
incentivized to do the same thing with our
customers.
Customers. So the reason we also care about
writing our own firmware, routing stack, all of
it,
is to make sure the operations and maintenance
are spectacular.
So the network can run really smoothly and
not be a challenge.
But at the same time, we're not trying
to take away any control from network engineers.
The software we invest in is heavily down
to being able to change, you know, like
an STP priority on a port all the
way to anything they'd like to change.
We believe in great software still for,
folks that are using our networks. So the
incentive question is something that we face all
the time in the network engineering world
in that
if you're incented
to sell, and then basically, I don't want
to say abandoned, but
kind of, it's not your problem, you're not
the one who gets the call at 02:00
in the morning, when the hardware or the
software fails,
your incentives tend to be far different
than if you are
actually the company maintaining. If you deploy it
and then you have to maintain it. And
this is a point of friction in operations
I see all the time.
And as a matter of fact, I see
it not just in the heart in the
hardware software world within the services world. Right?
Like, if you're a cloud provider,
and you want your service to be up
all the time, but this service availability number
is kind of an abstract thing that's out
there someplace.
It's not really me. It's like this out
there number. I'm I'm available 99% of the
time, whatever the number is, five nines, four
nines, whatever it is, doesn't matter. But
when my cloud service fails,
it's not me that's losing business.
It's my customers who are losing business.
It's my customers who get the call at
02:00 in the morning,
which leads me to build things.
I don't say cut corners, perhaps,
but to say
a lot more bailing wire and bubble gun
and duct tape.
Then if you actually were the person
doing the work of getting the call at
02:00 in the morning, because you know, your
customers down at two.
Well, my business hours are nine to five,
call me when it's 09:00.
Yeah, I've had that right.
It's interesting because when I was in Cisco
tech and escalation team,
many, many years ago, we never had that
attitude.
When I was in tech, that was something
that was great about the Cisco tech when
I was there. I don't know how it
is now. But when I was there, if
you took the case, you're on the problem
until it closed, and it was your problem.
And if the cast if your customer had
customers down, that was your problem, not their
problem.
Right?
You were on the hook to solve the
problem.
Make their network work again. That means shipping
hardware have at if that meant shipping hardware
and figuring the money out later.
That's okay, too. Like, oh, they need to
use 7200
to get them back on the air.
We didn't sit around and argue about who
was paying for it.
Right? You know, we picked up the phone,
shipped the box,
and then the account team could go figure
out who was going to pay for it.
That's that's not, that was not our problem
intact. Our problem was not the money side.
Our problem was to get the problem fixed.
So, yeah, so I think the networking industry
has changed a lot since then.
And nowadays, it's much more, you bought the
box, you figure it out, have fun.
You know, that's right. So I think it's
an interesting model that you're proposing,
just at least from an incentives perspective.
Now, from a network reliability perspective, something you
said is very interesting to me too,
that you're still trying to get deep into
features.
And maybe I'll push back on this a
little bit and see what you think about
it. To me, a lot of the problem
with network engineering today is we can do
everything 1,000 different ways.
And none of them are, quote, unquote, wrong.
I put the scare quotes around that, quote,
unquote, wrong.
I mean,
how do you maintain
a network
when I don't know, I walk into networks,
and they have five different tunneling protocols they're
using,
and five different ways of running BGP and
five different
IGPs.
And, you know, that impacts maintainability. So how
are y'all I mean, how do you think
about that? You know, just as a as
an engineer, how do you think about that?
Yeah. I think you require sort of two
seemingly dichotomous thoughts to be held together at
the same time.
One is you should be incredibly opinionated
about how things should be implemented
and go deep in that.
But two, maybe this is, to what you
were saying earlier, Russ, on how much the
networking industry has changed.
The industry has become
buzzword
alphabet soup
acronym amalgamation
the last two, three decades.
As an engineer and also the ethos of
meter is pretty simple on this, which is
we just wanna do the basics of networking
right.
And because we're building the entire stack and
not just like one small corner of it
and then throwing it over the wall somebody
else can maintain, we can do
protocols really well, implementations of,
implementations of these protocols really well and maintenance
of these protocols. And then we're responsible for
building the software all the way up to
the APIs and applications where engineers and our
customers will use them.
So we do believe in being really opinionated
on how to
implement these
things. But at the core of what meter
is is just doing the basics in networking.
Right.
And I know that's might seem
blase, but it doesn't seem like anybody else
is doing it these days.
Yeah. Interesting. Because, yeah, that's,
that's another issue. And part of what you're
also talking is like the whole idea that
it's not just about what protocols you deploy.
It's also about how those protocols are written.
But yes, I still go back to my
like, nerd knobs. Like, we gotta stop it
with a nerd knobs, the nerd knobs are,
are killing us.
Unfortunately,
I'm working in the lab right now working
on something for not not for my job,
but for something different. And I'm in this
lab work, and I'm like, trying to figure
out how to do something. And there's four
apparent ways to do it.
Yeah, but only one of them is going
to work. Correct. How many hours did I
go down trying to figure out like,
okay, that looks like it should do what
I think it should do. But it doesn't.
So maybe that does something different. So then
I gotta try to figure out what that
does. I don't think it's the engineers fault,
though. I think it's the vendors, because what
was happening to all these vendors is everybody
buys twenty, thirty companies a year, and those
are twenty, thirty different teams that are building
these features that are expected to work together.
And then by the time the customers and
engineers are using it, yeah, there are 20
different ways of doing it. There's no one
cohesive vision. It's not one cohesive product, not
one cohesive API and not one cohesive application.
This is like primarily what we disagree with,
how the networking industry has happened. Nobody really
builds stuff anymore. If you're a large vendor,
you're using to try to buy companies left
and right. And
seemingly,
this idea of I can buy two software
and hardware companies and say, combine,
that only happens in a business person's mind.
That's not how engineering works.
Yeah.
And even and even to the same point,
you're also faced with a situation when you
get to a certain size, where you have
customer a who wants to do x and
customer b who wants to do y.
And so when I worked at vendors at
Cisco and Juniper, and even
Erickson sausages,
old redback,
for those who know those those things.
There was a lot of pressure
to say,
okay, customer a really solve needs to solve
this problem, they would like to do it
that way. Customer b would like to solve
this problem, they'd like to solve it a
different way, even if it's the same problem.
These customers don't talk to each other. So
they're pushing you for two different features to
do the same thing.
And you're ending up going, well, which one
do I implement? Whoever's has the most money
on the table first, if they both have
a lot of money on the table, then
you implement both of them.
Even to the point where, when I was
at one company, we had competing account teams
going into a single account,
and trying to sell them to different vendor
products.
That's right. It's only gotten worse since you've
been away, by the way. It has gotten
worse to the point where customers are like
told not to buy the other thing by
the same company.
Yeah. Yeah. That's it is it's insane.
And it does impact
operations and serviceability.
So
let's go over a little bit of operations.
So the first thing with operations is meantime
between failure.
That to me is the first thing. So
there you're trying to address that, or you're
saying we should address that by just reducing
the diversity of operating systems stack software.
And
so that helps you to build higher quality
software.
Now there's a counter to that, which is,
what do I do about monocultures?
Because monocultures are a major problem.
I remember once we were asked to design
a network that would never fail by a
big bank.
We told them they couldn't afford it, and
they said try me. So what we did
was we built two network cores. It wasn't
their whole network, just their core.
We built parallel cores,
one running one routing protocol, the other running
another routing protocol, one running one piece of
hardware
with a one dot one operating network operating
system, the other running a different piece of
software, a different network network operating system. And
it was specifically because of situation they'd hit
was they had a monoculture,
they got it, they caught a defect, but
it was an all their package and all
their hardware,
and all their software.
And that costs like, a massive outage.
And they were very, very unhappy about the
massive outage. And I don't blame them because
it was on the network core.
So what do you think about monocultures
in relation to
trying to unify
all of this, everything into a single operating
stack?
Yeah. I mean, I think it depends on
how it's built and why it's built. And
is it built by,
sort of
trying to do some niche thing or working
backwards from what customers are trying to do?
We care a lot about mean time to
resolution,
not even mean time to failure. Like, how
do we resolve these things?
Then it's root cause for us on
where did the issue happen. Right? So there
are sort of three important bits in networking.
One is network design.
Second is network configuration,
then it's network maintenance.
So how do we get to mean time
to resolution?
Then understand where did the issue originate? Was
it in the design of the network? Was
it in the configuration of the network? Was
it during maintenance when we're making changes because
there's a bunch of new client devices that
are being added, security rules, where did it
happen?
So if you work backwards from understanding what
customer problems you're trying to solve
and build applications
like that, and then build APIs for those
applications like that, then build operating systems like
that, then build firmware like that, and build
hardware to service that,
then monocultures, I think, are less of an
issue. They are certainly an issue with how
the legacy companies do it.
Yeah, interesting. So you're saying basically,
the monoculture can be countered by
focusing on quality and focusing on
making sure you're doing the one thing you're
after
very well, instead of splitting yourself up and
doing all sorts of trying to do all
sorts of different things.
Okay.
So and you also said another interesting thing
there, which is that you're focused on MTTR
mean time to repair mean time to resolution.
So that is
another.
That's another area that people often don't think
about is that from an operational
perspective,
MTTR
is actually far more important than MTBF. In
most networks, most networks,
failures are actually,
I mean, I say, the law of large
numbers kicks in, right?
If I'm if I have, like, I used
to work on data centers that were, I
don't know, 10,000
1,200 routers, 1,300 routers, and therefore, when you
start thinking about 64 port routers
or switches, and you start thinking about 64
by 1,200, and you start thinking about optics
and
links and cables and
power supplies and all the other odds and
as they go into it, you can have
really high MTVFs on each of those components,
and you're still gonna have failures
constantly.
Like, two, three times a day, you're still
gonna have failures. And that's just gonna be
you have to work around it, in your
network design.
But
beyond that, like, MTTR
is very, very important.
And I think you're right in that having
a single user interface, a single
way to get into things, and a single
way things operate a single stack
is probably a very big boon
to MTTR
count, you know, once you get past the
monoculture
issues.
Now another part of that
is that
the interface.
So a lot of the problem we face
in the industry, and Greg Farrell was fussed
about this, and Ethan's fussed about it, and
I probably fussed about it at some point,
is that all our interfaces are different.
Like, s n m p MIB,
like, whatever happened to that, like, that's gone.
People still use it, but they only use
it for some things.
We're trying to move to Yang.
But most people just use Ansible modules, the
screen scrape.
And like we're never really I don't think
we're ever really going to get
to an operationally
secure and sound network,
unless we make the machine interfaces work.
It really worries me where we are with
this, that we're still
out here with Ansible modules that scrape five
different kinds of routers.
And
even for Yang, we're doing Yang conversion modules
and stuff.
And that that actually really
disturbs me. So any thoughts on that realm
of
how things go?
Yeah. It comes back to, again, what we
spoke about earlier, which is
trying to solve the problem
from a customer perspective, and then working backwards
from that.
In this case, what are customers trying to
do? Whether it's with Ansible or Chef or
Yang or Terraform,
anything,
is just to be able to do something
repeated
And then without having to know every single
minutia of commands and remembering that for every
single piece of hardware and every single route
and every single protocol,
that's what they're trying to get to.
We obviously started with CLI
in
networking,
then 20 ago sort of moved to dashboards.
We released a product called Command about a
year, year and a half ago now that's
gotten really great responses from customers and partners
that addresses
some of these issues of
being able to not needing to know every
single,
you know, little detail on a command and
things, but really being able to do automation
at scale and really being able to build
networks at scale. So I do think it's
really important.
I agree and I understand where your worries
coming from. But again, working backwards from what
customers are trying to do, and our view
is, you know,
being able to build
networks repeatedly of the same quality and of
the same
configuration,
and having to do it without having to
remember,
you know, every single small command line
incantation as possible.
Interesting. Okay.
So
another realm in which I think about operations
a lot that we don't address a lot
in the networking community or network engineering world
is just simply
testing.
And I don't just mean unit testing,
because of course, when I work for a
vendor,
and even now, I still deal with unit
testing versus system testing, and the combinatorial explosion
of system testing across multiple units.
So is that
something that you think about from an operational
perspective? Is this like,
again, just as an engineer, do you like,
think that this is an issue? And, you
know, is there solutions you think are coming
out there or
anything like
that? Yes. I do think it's an issue,
but I'm pretty sure,
there's not an engineer that's serious about their
work that does not think it's an issue.
But maybe you you'll hear me as a
broken record,
but we we do think here the integration
and the thoughtfulness from the ground up matters.
Here's how we do it, and maybe to
give you an example,
I know this is a,
this is sort of a audio only version
of of this of this talk, but, you
know, happy to show Yeah. Specific instances at
some point. But
if you think about how networking started, most
of the intelligence on how to do routing,
what the configuration is, how to test the
network was basically all inside the hardware as
much as possible.
Then obviously, the buzziest thing that happened in
networking twenty years ago or fifteen years ago
was software defined networking.
And
the idea was to move a lot of
the definition of this networking into software
so that it's malleable
to do it.
What I think we've done over the last
five, seven so five or seven years is
a little bit
interesting and sort of next step after that
is we moved
all of the intelligence
of the of the network actually into our
back end down to a single port.
What that means is we can actually configure
thousands of pieces of hardware down to the
protocol level, how they're connected, how
the topology is,
make sure the configuration is right on these
networks.
And only
when we are happy with it and a
physical hardware is connected,
the hardware downloads the source of truth from
the back end, and then it's off to
the races. But what that enables us to
do
is to be able to configure and test
these networks
virtually first.
And then when the hardware is plugged in,
these virtual networks are the source of truth
that come down into the hardware.
So it's a remarkable system that's enabled us
to scale a lot, and it's one of
our sort of
differentiating
ways to make sure networks are configured properly
because configurations are where majority of the issues
happen and then being able to test these
networks and then maintain these networks. So we
moved everything
into a virtual network which is how all
of our networks are first configured.
Then when the hardware is plugged in, it's
simply a mirroring of that virtual network. So
that's been our solution for
it. Okay. Interesting.
So but in the larger network world, you
would say that it's still a concern. People
are still very worried about this.
And,
yeah, and it's still an issue
that needs to be addressed in some way.
Now another thing that I often think about
is in terms of operations
is lifecycle.
Another thing that we pay zero attention to
in the network world, I have
seen routers handed down
from place to place inside an organization
and use for five different roles. Because once
you buy it, you own it. Now I
know your business model is a little bit
different than that. But do you think that
there's
what would your advice be to a network
engineer
beyond just,
you know,
switch to our model? Is there any other
advice you could give network engineers about that
problem? Because
I don't I don't know how to fix
it personally other than go to my manager
and say,
just buy new stuff.
Stop it.
I think, network engineers can make finance teams
and procurement teams their best friends.
And then that's how it can happen.
Or switch to METERS model. Sorry. I had
to say it. Yeah. That's fine.
So when you talk about making that, you
mean, like, go play golf with them.
Yeah. Take them out to dinner, make sure
they know how important the network is. Yeah.
Make sure they understand that everything runs on
the network.
And if the network isn't great or the
network isn't secure,
how detrimental it is for the entire organization.
Because I think a lot of times these
days especially,
people don't realize the importance of networking,
and it's sort of an afterthought. So I
do think there is sort of a shift
in the network engineer's job of making sure
people actually understand.
It used to be that everybody
knew the network is the heart and
soul of it. But I don't think that's
true anymore. And I think it's the job
of the network engineer to make sure everybody
understands.
Yeah. And I think that's part of it
is that
we actually play against ourselves when we call
the network a commodity.
That that actually hurts us.
Because
people say, well, if it's a net, if
it's a commodity,
I just want you to add more bandwidth.
It's like adding memory, right? You just add
more bandwidth, like it's, it's not a bit
just add bandwidth,
you know, or just just make it faster,
like it's a commodity, I can make a
car that goes faster. So just make it
go faster.
Like, it's actually not like that.
Routers and switches might be commodities.
Fine. Got it. They're cattle, not pets.
The network,
in my opinion, is still not a commodity.
The network is a system that is built
up of commodity pieces.
But that doesn't mean the network itself
is a commodity.
So I think we confuse this, and we
work against ourselves, and we say these things,
and then we need to realize, you know,
we need to change our language.
And the way we approach
the company management, corporate management, and financial people,
to help them understand that you can buy
commodity hardware, fine, whatever, don't care. But your
network is not a commodity.
You need to take care of the network
as a system. And that's still going to
cost money. So that's, that's kind of a
common approach for me, at any rate,
is to do it that way.
Yeah, make sense.
So
I don't have a whole lot more topics
to talk about. I mean, I have a
couple but I don't want to dive into
yet another half an hour's worth of discussion.
So we should just wrap it up there.
Anything you want to say before we jump
off, Anil?
Networking is important. More people should study it.
Networking should be cool again.
Yes, I agree. Yeah, we're definitely in this
mode where
we don't have enough network engineers coming into
the into the market.
I look, I've been doing this for thirty
years, I suppose, something like that. I look
behind me. And not only do I not
see a lot of people coming in, oh,
that's not absolutely true. You know, I teach
a couple of classes in network engineering, and
I know people other people who teach classes.
So I know there are people out there.
I don't know if there's the quantity that
we need.
And to be honest, not to throw shade
on people,
I think the way we teach networking now
actually develops people in the wrong way.
Like,
I'm actually write a post about this at
some point about teaching from the outside instead
of the inside out, and how we teach
from the outside in, and how much I
think that's a problem.
So, yeah. Alright. Cool. So,
Neil, where can people get in touch with
you if they want to? Do you have
a blog? Do you
LinkedIn? Do you what do you do?
Sure. LinkedIn.
Our website is meter.com,
metr.com.
My email is pretty simple. Anil@meter.com.
People can just email me. Okay. And but
you don't have a blog or anything like
that. You don't do any writing. No. Alright.
Just we just write on the meter blog.
Okay. Cool. Alright. And I'm Russ White. You
can always find me here at the hedge.
You can find me at rule eleven dot
tech.
You can find me on LinkedIn. You can
find me on x, me, Twitter, whatever you
want to call that thing. I'm actually really
easy to find. If you have topic suggestions
for the hedge, please let me know or
people you'd like to see on. And we
know we live in an attention driven economy.
So you spending the time to listen to
us all the way to the bitter end
is really important and we thank you for
spending the time with us and we will
catch you next time.