Join us at The Hedge for a conversation
about engineering,
technology, and business. In this episode, Jeff Houston,
Russ White, and Tom Ammon dig into the
first part of their discussion of DNS over
HTTPS.
Good evening, Jeff. I think it's morning for
you in Australia because you're upside down.
It is morning. I'm on my head, but
the sun is out. Yes.
What you see above, Jeff, which you can't
really see is actually
the floor. Just don't tell anybody that.
Tom, how are you tonight?
I'm doing fine.
Good. Good. So
This on this episode of the hedge. We're
here to talk about dough
If you say it real fast 10 times,
you'll understand what it means
dough
DNS over https
and, Jeff wrote an article in the Internet
Protocol Journal, which was really really good. And
by the way, if you don't read the
Internet Protocol Journal, you should
it's an occasional. I don't know that Olay
does, I'm not sure how consistent Olay publishes
it. But when it does come out, it
generally has very good articles in it like
this one by Jeff. It's well worth reading.
So, Jeff, talk to us about dough a
little bit. Tell us what it is,
And then we can back into trying to
understand some of the architectural
issues and stuff that's going on there
Okay, you know, thanks Russ and, good morning.
You and John
Look,
it's one of those really simple acronyms. It's
DNS
over
HTTPS.
Now you kinda go It's really Doze.
Doze, I suppose, if you stress the s.
But these days, HTTP is so yesterday. None
of us are allowed to use it. So,
you know, as soon as you say HTTP,
you mean HTTPS.
That sounds innocuous,
and
to some extent, it is.
You might recall various little tools and techniques
to get past firewalls going back almost twenty
years,
where we actually managed to get IP itself
stuffed into
HTTPS.
What's that? IPH, I suppose, is the way
you'd like to call it. And doing the
DNS in HTTPS
is certainly nothing new.
And and to that extent, it's sort of,
so what?
But
it became an RFC
really quickly,
about a year ago, 2018.
And the IETF
meeting in March
2019
had a session to
talk about dough, you know, I think we
need to talk about dough,
and the room was certainly packed.
And the conversation was,
highly tensioned.
There's a lot of feeling about dough.
And So it's What's the problem we're trying
to solve? I know one problem is
people complain about middle boxes,
messing around with DNS,
and another problem people are fussing about is
man in the middle attacks against dns. Right?
Those those are kind of
Those are my impression of where the problem
people. Well, we have to go back to
the the I think for the I t
f the snowden leaks and those allegations were
a defining moment for the IETF.
I'm like, they'd had their moments before when
they were asked to standardize various, eavesdropping protocols.
And, you know, some of the really old
gray beards might remember the CALEA fuss.
They are whether the IETF was gonna work
on standardizing,
tapping protocols in in the media gateway controllers.
And it goes back further even to Raven?
Yeah. All of those old debates. So so
up comes
up comes Snowden.
And what emerges, of course, is this huge
amount
of
state surveillance,
in amongst and trying to find some oxygen
from all the commercial surveillance that happens anyway.
But some of the reactions to this
was a long hard look at the IETF
standard protocols
with the view of saying,
if you were a reasonably
resourced observer
and you had access
to wires, to routers, whatever, you had access
to infrastructure,
how much could you see with not a
lot of effort?
And, of course, the results are quite surprising.
If you're using unencrypted
HTTP,
there's a huge amount this is sitting there.
But equally, if you're using the standard DNS,
which is open,
unencrypted,
runs on UDP port 53,
then you can see all the queries and
all the answers. You kinda go, well, boring,
not boring.
Almost everything that you and I do
starts as a name,
even the ads. And your machine
translates the DNS label, or part of that
name, into an IP address and then goes,
well, okay. Make a connection. Off we go.
Let's do fun stuff.
So if I had
an up to the second
record
of all of your queries
as you make them,
I know what you're doing.
I know who you are.
I think I can put you into a
profile.
I think I understand you.
And if I work just a tiny bit
harder, I think I know what you're gonna
do next.
Now that's kinda chilling in so many ways
because all you're doing is just playing on
your computer,
yet the sort of data exhaust, the fumes
that come out of your digital wire
are astonishingly
rich in information.
And so
the the move was on at that point
to harden up our protocols and one of
the places we wanted to harden up, was
the DNS itself.
So the sort of Uber the overall project
was called DNS privacy,
and there are a number of approaches that
have been made to try and improve the
DNS to make it not impossible,
certainly not impossible to look at, but to
make it harder.
There's no absolutes in this world, but if
it costs you a large sum of money
to see what I'm doing, you won't do
it casually. You might still do it, but
it won't be a casual decision. If it's
gonna cost you a lot of money to
find out what I'm doing, you're gonna figure
out whether you wanna do it or not.
And that's kinda where this work started.
So
the first thing was, well, what can we
do about the DNS? Well,
it's an awfully chatty protocol,
but it chats in the clear.
So can we
kind of make it chat down an encrypted
wire?
And the answer, surprisingly, or maybe not so
surprisingly, is sort of, you know, yeah, we
can.
And this started
the entire issue of DNS over something other
than open UDP.
Now
right at the word go, the DNS actually
had two transport protocols, TCP and UDP.
We used UDP because it was fast, efficient,
lightning quick, just the thing you need for
the DNS.
Oddly enough, it's also really easy for a
third party to manipulate,
alter,
fake,
and generally make a menace of themselves because
it's UDP.
I send out a query to a supposed
IP address.
I get back an answer
from a supposed IP address.
Is the machine
that is faking the answer the machine I
sent the the query to? I don't know.
Is this a real answer?
I don't know.
Should I trust it?
Well, everyone does all of the time. Well,
doesn't doesn't DNSSEC solve that problem, though, that
particular problem?
Look. If if everyone signed, if we're in
the room of DNSSEC,
I would say that's a whole lot better
than the room we're in now,
but no one signs their domains with DNSSEC.
So an awful lot of focus sitting there
with the magnifying glass going, if you sign
it, I can check it.
But if no one signs so what well
and beyond that even I think even going
to that point Even if everybody did sign
you still could do man man in the
middle attacks from what I understand well
Okay, this is one of these but you
could
But this is one of these age old
tensions between security and speed.
Right.
Over in browser land,
they talk about the billion dollar millisecond.
And at one point, you know, one of
the browser folk had even toyed with the
concept of putting the DNS right inside the
browser because it would just shave off milliseconds.
They do all kinds of weird and wacko
tricks.
If you want to validate your DNS responses,
you better bring a packed lunch
and possibly a few supplies because your web
experience
is about to get an awful lot slower.
And and no one's willing to go there.
Right. So, you know, you might say, well,
let's all use DNSSEC.
But in a world where one second is
regarded as an as an eternity,
you know, just no one's gonna do it.
So,
okay, we need something different. We need something
a bit better. We need something that's fast
and certainly no slower than the DNS as
we understand it,
but doesn't kind of broadcast what you're doing
to your neighbors, your ISP,
and and various other folk who have an
interest in seeing what you're doing.
And the approach started,
I suppose, innocuously,
like all things.
You have to break down the DNS into
two parts of a conversation.
There's me and you out at the end.
And pretty typically, when we turn on our
machines,
our infrastructure, the normally, the Internet service provider
says, hi. Here's an address for you to
use. Here's you know how to get started.
By the way,
I've rigged up some DNS resolvers, what we
call recursive resolvers.
Load those IP addresses into your machine, and
whenever you have a query, just ask us,
and we'll give you the answer.
And so that first hop
is called the stub, me,
the stub resolver, the endpoint,
to the first piece of DNS machinery
that's doing the hard work, the recursive resolver.
Most ISPs run recursive resolvers, and there are
a few other big companies out there.
Google springs to mind.
Cisco bought OpenDNS.
There's Quad nines. There's,
Cloudflare.
There's a bunch of these folks that also
run
open recursive resolvers.
If you just do nothing, you're probably gonna
run with the resolve that your ISP gave
you.
If you make an effort, you can go
and use someone else's.
Now the next step, recursive to the folk
who really know the answer, the authoritatives,
is not part of what we're talking about.
So I ask recursive. The recursive asks the
authoritatives.
Let's just ignore that second part of the
puzzle.
It has its problems, but let's not talk
about it today. Let's talk about the bit
that matters,
the bit where you know it's me,
where Jeff is making queries, and it's pretty
obviously Jeff.
And what we're trying to do is to
stop other folk from looking at Jeff making
these queries.
So what we'd like to do is encrypt
the session between my machine
and that recursive resolver over there.
Now
we spent a lot of time
making sure that the web was secure using
a thing called transport level security, TLS.
Yep. It started off as a secure socket
layer and then sort of morphed by name,
but it's it's actually the same old stuff.
And the way it works, of course, is
when I open up a secure session,
I first start in the clear
and say, hi,
Russ. Good to see you.
I wanna connect to rus.net.
And you go,
okay.
Here's
a key
for rus.net,
and here's my certificate.
And I go, well, fascinating. Why should I
trust this random digital stuff?
But luckily, I've got inside my machine
about 1,500 trust points.
And if I can make a secure
chain of signing
from one of my local trust points
to the digital key that you've offered me,
I believe that certificate, and I believe it's
you, and I'm now talking to rush.net.
You kinda go, well, so what? What about
my Internet banking?
You know? I'd like to know I'm talking
to the right person. Offer me a certificate.
But that's not all. Because at that point
then, the two of us do a very
rapid exchange and create a session key,
And then all further communication
is encrypted using that session key.
The beauty of the session key is even
if you know the public key of rust.net,
it won't help you. And even if you
know my public key, it won't help you.
To actually break into that session, you need
to have knowledge of private keys. And by
definition,
if you know my private key, you're me.
So TLS, as far as we understand, and
and with some caveats, TLS is a pretty
good protocol.
It better be
because
there's nothing else in the cupboard.
Yeah. I was gonna say that's pretty much
all you got. Right. It's kind of what's
security on the Internet? TLS. Yeah. Well, what
if TLS doesn't work?
Jeez. You shouldn't have asked that question because,
hey,
no answer.
Right?
That's it. That's that's that's you have to
live with it.
So we wanted to encrypt the session between
the stub and the recursive, and so you
reach into the toolbox and you go, TLS.
Now, TLS is a TCP protocol,
not UDP,
because this security association takes a little bit
of handshaking,
and there's kind of a shared crypto state.
And while some folks spend a lot of
time and a lot of effort trying to
make
UDP
TLS work,
DTLS, Datagram TLS,
the results have been largely paperwork.
As someone once described about the old ancient
OSI protocols as vaporware about paperware, or was
it the other way around? Paperware about paperware.
DTLS is largely the same kind of, you
know, e.
Looks fine on paper,
and that's all you get. So, okay, we're
down in TCP land.
Now this DNS over TLS excited some reaction,
and one of the first of these was
the DNS will melt because there are literally
trillions of these tiny transactions every single day.
And if all of the servers had to
run TLS,
all the recursives instead of UDP,
obviously, they'd all die and go to silicon
heaven.
That's an odd statement to make
because
we use the web
like crazy.
And there are just as many short web
sessions probably as there are DNS queries and
answers. I was gonna say they probably were
not paying attention to the work done around
ILMP,
for instance.
Right. So we know how to make
massive TLS work. Right. Right? Because the web
is living proof that trillions of sessions a
day Right. And not work. Very fast DNS
to work at the same time. Yeah.
Okay. So
DNS over TLS
looks pretty good, and and it kind of
does a couple of things which are not
bad at all.
I know who am I talking to.
If I say I'm going to CloudFlare, if
I'm, say, the quad nine, I get back
a certificate.
And so my friendly local spook agency
cannot fake it out unless they've been playing
with my machine.
Unless they've done the Kazakhstan thing and inserted
a fake certificate on my machine,
I can't be fooled.
Now, okay,
install faxes it, you're dead, fine, we all
know that.
But on the assumption,
my machine is roughly as, you know, I
configured it and I have trust points that
I am prepared to trust, then I'm talking
to whom I want to talk to and
no one else can see it while I'm
talking to it. So that traffic is a
secret between me and my chosen recursive resolver.
This sounds great.
No one cared.
And, again, this is kind of moving towards
dough. Why did no one care? They're geek
knobs.
So there are, what, four and a half
billion users out there.
Roughly, how many are gonna configure DNS over
TLS into their machines?
I've got
10 figures on my hand. Okay. I'll go
10.
Yeah. That's because, honestly, I don't. I don't
know. You know what? It's hard, and it
doesn't quite work. And and and and so
it largely remained a paper exercise going, we
could do this. Aren't we clever?
But as a tool for the masses,
nah. It was never gonna happen. So everyone
just yawned.
Now
the move from DNS over TLS to DNS
over HTTPS,
in protocol terms,
is about one millimeter to the left.
You know, if you think about it, it's
still TLS.
I still know who I'm talking to. It's
still encrypted.
And inside DNS over HTTPS,
the packets are actually wire format DNS,
which is the same as HTTPS.
So
nothing's changed.
You go, well, what's the fuss about?
Okay.
Next next issue.
If all you're doing
is playing with the operating system and changing
the way your operating system talks to the
recursive resolver,
so what?
But the folk who really took this up
were the folk who deal in HTTPS,
otherwise known as the browser people.
Now the browser people have a number of
motivations.
And in some ways, they're very worthy motivations,
but they have their own motivations.
They don't trust the platform.
If you're Firefox,
you're my customer. You're Firefox's customer.
You're me, and I don't really care what
the underlying Android or Apple or anything else
is doing.
I, Firefox, am here to protect you.
So I want control.
I want to make your browsing experience one
that I control.
I don't wanna leak.
I don't wanna gratuitously
send out information,
and I wanna be fast. I want control.
So you've offered me a way to do
DNS over HTTPS,
and I am an HTTPS engine.
Well,
marriage made in heaven. Let me do your
DNS for you.
So then why not just stick the resolver
in the browser itself? That's what we're talking
about. So that's what the whole angst is
all about.
The resolver gets stuck in the browser. Now
the browser
now has a knowledge of the DNS
that no other application
on your platform has.
So your browser might know, I'm going to
rust.net. Here's its IP address. Blah blah blah.
I've done a web fetch.
But if you bring up a terminal window
and go SSH rust.net,
the operating system goes,
because it's not a common pool anymore.
Right? So the application
is now making its own choices.
Now
which resolver are you gonna go to? Because
you're running HTTPS, and like the web, you
can go anywhere.
And so the ISP says, hi. Here's some
friendly resolvers.
Now you could be in China.
You could be in Kazakhstan.
You could be in many places
that offer some kind of DNS censorship for
free.
Even The UK and Australia, it's just part
of the service.
We care. It's free.
But if I'm a browser
and I tunnel across to
a DNS provider that says,
I will translate anything that's in the DNS.
I'm not going to censor you.
I'm making an independent decision as a browser.
Again, though, let's look at this a little
bit more. If the browser is doing it,
then it's no longer a nerd knob.
So let's say,
just for one
microsecond,
that Chrome decides to do it. And let
me point out they have not decided to
do anything yet.
But if they did,
78%
of the world will then do it
because that's its market share.
So if Chrome decided to take all of
its DNS
and quietly direct it to a trusted recursive
resolver
that Chrome has chosen,
no one else would see it because it's
encrypted.
No one else could argue with it because
Chrome did it, and it would be hard
to undo it because it's a nerd knob.
And it would break all this and it
would break all the CDNs. Right?
Well, break a whole bunch of stuff.
Not the least national policies, but CDNs, VPN,
you you name it. A whole bunch of
things would get broken that kind of assume
the DNS works one way.
And these guys are tunneling
from the application itself
across to some other point,
a point that
the user, the platform, and the intermediaries
have no say in.
It's the browser's choice.
Now on the browser side, they kind of
argue, look.
You got caught with your fingers in in
the till. You got caught looking. You got
caught abusing us.
We don't think you're worthy of the user's
trust anymore.
We, the browser folk, whose interests are you,
the consumer,
think you deserve better than this. So we're
gonna take the decision away from you and
basically put you in a different room where
your DNS doesn't leak.
So this is the kind of the pros
and cons of the argument flying around here.
The some folk kinda go, well, we're used
to the DNS
being
an open piece of infrastructure.
And whether you're a CDN provider or a
VPN provider,
whether you're a platform provider,
like Android or or Apples with their Mac
Mac OS's and iPhone, if that's your stock
in trade, then
you do the DNS.
And if you're an ISP, you do the
DNS.
But the browser folk are kinda going, with
DNS over HTTPS,
I don't necessarily have to play your game,
and I can use defaults
inside my application
that lift the application
away from you
and take it out of your control.
Right.
And at that point, you know, we light
the firecrackers.
We take a step back and watch the
light show. Yes.
Well, right. And it and it becomes I
mean, there's just so much about this that
is
potentially troubling. Right? There's so much about this
that could just that could go wrong. Not
that it necessarily will, but it could.
Well, let let's talk about this, and let's
actually go there because, you know, why not?
There is no doubt
that the underlying certificate system that the DNS
uses as its
trusted rock of security
is pretty corrupt and pretty horrible.
We all wish that were otherwise,
but
folk who
get listed
inside software as a trusted,
issuer of certificates
are sometimes corrupted by, you know, mere money
and do things that they should not do.
And the problem is you can't tell a
bad certificate from a good certificate. They're all
just certificates.
Symantec,
which is now no longer in existence,
got caught certifying
the domain name that doesn't exist, example.com.
Now
if I tell you I'm a certificate authority
and I hardly ever lie,
and I'm not gonna tell you which ones
I'm lying about,
you can't trust anything I do because I
could be lying for just you. You know?
And this is the problem that kind of
besets
the certificate world.
There have been numerous approaches to try and
clean this up.
And do you know the best answer we
have?
It's an old one, actually. It's called certificate
transparency.
The best answer we have
is that a trusted certificate provider
has to publish in the open
every certificate it issues.
So the only thing that keeps the CA
system from going
completely off the rails
is openness.
Now
I can still be bad,
and I've got about, I don't know, ten
minutes before you'll notice.
And and it's not a very good defense.
But after about ten minutes, someone might see
this logged fake certificate and go, that's bad,
and we'll go and fix it up.
But the point is, well, the only thing
that's keeping all this stuff together is openness.
Now the DNS
was an open protocol.
You could see it. Anyone could see it.
And that was both its strength and its
weakness.
Because if
I started to stand up a recursive resolver
that lied,
literally lied.
You wanna go to facebook.com?
Try, oh, 1.1.1.one.
That's a lie.
Well,
I can see it's a lie because it's
open.
But what if your browser does it through
an encrypted channel
and the other end of the channel lies
back? No one can see it.
So
one of the things we lose with DOH
is actually the openness of the namespace.
Now
we talked a little bit about CDNs and
VPNs and customized spaces.
But what if
I set up a trusted
DoH server
that deliberately
creates a so called fake name space that
is tuned against your application?
Good or bad?
It's kind of taking the split DNS model
and making the split not obvious.
Join us in the next episode of The
Hedge as we continue our discussion of DNS
over HTTPS with Jeff Houston.