Geoff Huston 0:00 Yes, start at the very beginning. Why does the DNS have a root zone? And there are many ways to design information systems purely distributed where there is no single point of authority, otherwise known as an anarchic mess that doesn't work. Or you can see my bias, or you root things in a strict hierarchy, and so every point in your network is the descendant of something and has its own descendants. And if you follow the chains back up, you get to a single point of authority. The DNS is structured like this, and if you even see it in the names, www.apnic.net, Dot. George Michaelson 0:56 you're listening to ping, a podcast by APNIC discussing all things related to measuring the Internet. I'm your host. George Michaelson, I'm talking to Geoff Huston from APNIC labs again in his regular monthly spot on ping. This time, Geoff and I talked about the DNS again, but it's a slightly different conversation to the one we usually have, which is about the DNS protocol. The DNS exists to tell people how to find things, but over 50% of the traffic seen at the root of the DNS is a question which can only be answered by no that doesn't exist, and it has been even higher alongside this massive rising tide of negative answer questions. There are concerns about how the root zone is served and why we serve it at all. Geoff has been looking into the numbers here, and he thinks we have a path out which leverages a new zone MD record and a shift in behavior around serving data of the root and how it's fetched. Geoff, welcome back to ping. What should we talk about this time? Geoff Huston 2:07 George, it's an absolute pleasure to be back on ping, and we're going to talk about something that almost unintentionally has consumed a large amount of my life over what seems like an eternity, but in actual fact, has only been about the past five years or so, and that is the root zone of the DNS. They kind of go but Geoff. Everyone uses the root zone. It's just the name space. It's just, yeah, that's why we live with it. What's that to talk about? There for my sins, of which there are many I believe George Michaelson 2:37 Not on this program. This is a sin free zone. Geoff Huston 2:41 Oh, okay, well, behind the curtain are a bunch of sins of my younger self, and as penance, I'm actually on two committees related to the carriage of the root zone of the DNS that are kind of facilitated, although it's a community effort by ICANN, the folk who notionally look after the policies around the name space, new TLDs, etc. One of those is the root zone evolution Committee, which, although it's been a very grandiose title, is actually a sort of forum to discuss ideas about what's in the root zone, and how do we get things into it if we think it's a good idea, and how do we get if we think it's a bad idea? George Michaelson 3:24 So that's two slightly different topics, in a way, isn't it? We've been used to talking about DNS as a protocol and about the protocol behaviors of packets flying over the wire, and we've been talking a little bit about signing things and the importance of cryptography and chains of trust and the descent down. This is slightly different Geoff. This is about what things are in that zone and why are they there. Geoff Huston 3:51 Right now, you kind of go, Is this a thing? You know, initially the root zone had an authority record. This is the date and the version of this particular incarnation of the root zone. Things change over time. Here's the list of labels. Here's the name server, names of those delegated labels. And when we introduced DNSSEC, we added some keys in. What's your problem? Well, the latest one that went in by courtesy of this committee and an RFC was a weird little entry called zone message digest, which was the cryptographic signature of the entirety of the zone signed with the key signing key. George Michaelson 4:33 Now, just in case there is anyone on the planet who hasn't listened to ping before, records have types in the DNS. And we're used to the record type, the A and the quad A. That's how you find the v4 address and the v6 address. And we've become used to the idea of the key record type and the signature record type, zone, MD. That's a new type of record, isn't it? Geoff Huston 4:56 It's a new type of record. And interestingly, it's not. About any label in the root zone, or any zone, really, it's about the entirety of the zone sorted in, you know, a canonical order. And it's a cryptographic signature that only matches with the key signing key if you've got the right root zone. If someone just invented a root zone of their own, and said, Well, here's a key. And the answer is, well, it wouldn't compute unless you had knowledge of the private key of the KSK, which only the root zone operator has. So it's actually really quite a neat little facility that doesn't talk about delegations. It talks about, is this zone authentic and current? George Michaelson 5:40 This is not in transport, not in protocol. This is the behavior that is classic cryptography. Know one thing outside of the DNS have trust, because you get one piece of information out of band, and using that one piece of information, you can now validate things that are said to you if it lies under that chain of trust. And this one thing you said the word canonical. So it's organize the zone into a very specific order based on highly prescriptive rules. Get rid of weird spaces. Use specific alphabets in Unicode to represent it. Do a thing, and then compare what you did with what you receive in this signed object, and if they're the same, it means that the zone you're looking at, that state of the zone, is absolutely the same as the guy who signed it. And you've just said, the only people who can sign it are the people who make the zone. Geoff Huston 6:38 Right. It's almost in the real world, where you go up to a notarized lawyer, a notary, and say, here's a document and here's a copy of that document, dear notary, please affix your signature to the copy to say you've looked at both, and you notarize that this copy is an authentic, genuine and accurate copy of the original. And evidently, there's a register of these notaries. So I can send this copy to anyone. They can look up the register of notaries, and if signatures match, they can go, yep, Geoff's not lying. Same thing digitally. But that was the first of my sins, George, this root zone, and then we'll get back to zone MD, because it's actually relevant to this, this discussion. But the second one, and this was a whopper scene, whatever thing I did, the dear old IAB the Internet Architecture Board nominated my name forward to be a member of a sort of a weird little committee, the root zone governance Working Group. [George: Hmm], we have to sort of backpedal and then go forward. So I am going to backpedal a little bit and talk about the role of the root zone, and talk a little bit about some of the dynamics, and then talk about the role of this committee, which will then sort of land us squarely in what I regard as my personal penance for a life misspent because it's not a fun committee. George Michaelson 8:07 Start at the very beginning. Geoff Huston 8:09 Yes, start at the very beginning. Why does the DNS have a root zone? And there are many ways to design information systems purely distributed where there is no single point of authority, otherwise known as an anarchic mess that doesn't work, or you can see my bias, or you root things in a strict hierarchy. And so every point in your network is the descendant of something and has its own descendants, and if you follow the chains back up, you get to a single point of authority, right? The DNS is structured like this, and if you can even see it in the names, www.apnic.net. DOT. George Michaelson 8:57 so, that.on the end is that weird moment that you don't have to have it, but it's kind of implicitly there. It's the real.at the end of the sentence. And that.if you don't say it, it still kind of exists. Geoff Huston 9:12 Yes, www is a descendant of APNIC. So you'll find a definition of www in APNIC. APNIC is a descendant of net. You'll find a record for APNIC in the .net zone. Net is defined in the top zone, the thing at the apex, the root zone. And you kind of go, Well, okay, that's the structure of names. Unlike a dictionary, where it's just a collection of names in some order, every DNS name sits somewhere in that hierarchy, and every valid DNS name ends with a dot George Michaelson 9:47 for people who have old memories. The pre existing state is that there was a file that was distributed from a single point in the states you fetched over FTP and. That was called Host dot text, and it did not require names to have any form. And it didn't even really require names to be unique, which is kind of a bit weird. And you were free to edit that file and remove things or add things, because the behavior here was not in any sense structured. You could have that file, or you could not have that file. You could make your own. There was also a system in UUCP, the old dial up, store and forward network, that created domain names and allowed you to map weirdo host name strings that were just names into a structure. But that structure was also not very centrally policed. You could intrude names of any kind in there, so you can have a very anarchic situation. But this one, this one is very definitely structured. Geoff Huston 10:55 It's a bit like giving everyone a dictionary and a pencil and an eraser. [George: Yeah], you can erase some entries and go and write your own. But I tell you, what, if you invent a word in your dictionary, it doesn't mean anyone else understands it. And the whole thing about giving everyone a copy of a common file, doing changes, well, you can change your local copy of host, dot text, no one else is going to believe you haven't told anyone else, [George: yeah]. And the whole thing about names is that they are common references, [George: yes] so you can change your own dictionary, your own meaning, but doesn't mean diddly to anyone else. [George: Yeah], you've got to kind of globally change things. And when the DNS fact, when the Internet grew beyond a few 10s of machines, and they actually thought, Well, we actually need a system that's distributed yet coordinated. The namespace was accompanied with a common distributed database definition and a lookup protocol. And collectively, we refer to the DNS as the whole box of bananas, the space of structured names, the protocol that, if you will, wanders through this distributed database, and, of course, the distributed database itself. George Michaelson 12:12 So you get all three in one go. You get a protocol to find things. You get a hierarchical structure that's understood with syntax and behavior, and you get a really strong assertion. We need to have a unitary route. This is going to be the anchor point. If you take this box, you get everything lie under this single anchor. Everybody agree to that. Geoff Huston 12:35 Yep. And at the time the Internet was small, the number of people had to agree to this was vanishingly small, even so, I have to say, the UK never agreed for years. They always did things. [George: Ah, yes], but the rest of us, George Michaelson 12:49 so you have sins Geoff, and I have sins, and for my sins, for my sins, I was a member of that community where a great phrase was said by somebody in the community. "It's an experiment". Let's do things the other way around to everyone else in the universe, because "it's an experiment". Geoff Huston 13:11 Uk.ac, dot, Janet, as I seem to recall. Yes, Well done, everyone. But we digress, and that's fine to digress, but let's, let's get back again to this distributed database. Not everyone knows everything. And by that, I mean there are a number of DNS authority servers out there that understand all about the zone, apnic.net and they actually have an entry in there for www.apnic.net, [George: yeah]. And if you ask that machine, it will say the IPV four addresses. This the IPV six addresses that it's DNSSEC signed, and so on. Ask it the right question, and it will give you answers. But I've just booted up my machine. It's an innocent child in the Internet. How does it know to ask that authoritative server? George Michaelson 13:57 Yeah, finding the authority Geoff Huston 14:00 finding the authority is a much harder problem because it doesn't know. And so this is where the structure of the DNS comes to your help with, oddly enough, the awesome power of recursion, because the cut through model. And for this, I'll dip my lid to Paul Mockapetris, it was a brilliant piece of computer science. Is that you start with a known anchor point. Let's call that the servers for the root zone, and that you pre load as part of the config. Here you are. Here's some folk who know everything about the top level domain in the root zone, George Michaelson 14:36 but that's all they need to know. They don't need to know anything about longer names down in the edge. They just know that top layer Geoff Huston 14:46 that top layer, those you know, 1400 or so names that are sitting in the root zone. Now what you do is you ask them the standard question, hi, I want to know dub, dub, dub.apnic.net, ipv4's address, and its answer is, well, bud, you're really asking. I know this stuff from dibbly. I just know the 1400 or so top level names in the root zone, but I'm going to get you back a referral response, which is, this is not the answer, but this is somebody, or a set of folk, who might help you with finding the answer. It's called a referral response, and it's actually going, here are the name servers of.net [George: Yeah], maybe they can help you. And then you take that answer and apply it the same way. So now you have a bunch of name servers, George Michaelson 15:39 same question, same behavior, different party at the other end, but you're behaving exactly the same. Geoff Huston 15:46 And so you repeat the question, but now you send it off to one of the servers for.net go. I'm searching for the IPv4 address or v6 so I'm very switched on and current apnic.net and the name servers for.net go again, son, I'd like to help you, but you know, I'm not the body you need. But here's another referral response, because APNIC is a delegated zone, and here are the list of name servers for apnic.net, whatever they might be, knock yourself out. At that point, I do the same thing again. I take this referral response and go, Hi name server for apnic.net I want to know dub, dub, dub. And the answer is here, here's an answer. And we all go away singing and dancing and, you know, prancing happily through the fields. Now, this works badly. George Michaelson 16:33 Well, it does work. Geoff Huston 16:34 It does work. But how many users are out there on the Internet? A few billion or so. How many domain name queries are being made, oh, a few trillion or so. If every single query started at the root zone, this would never have taken off. It would have just melted before it even started. This does not scale, George Michaelson 16:53 right, because every single question in the world winds up going through the root in order to be answered. So there's no point of efficiency here. You always wind up asking at the root in this model, as you've described it, Geoff Huston 17:06 Right? So it doesn't scale. Oh, by the way, it's a massive privacy leak, because those root zone, that root zone server, let's just say, for argument set, there's only one knows what everyone is doing all of the time, the Panopticon of the DNS so a it's a massive privacy leak, and even, I suppose, more critically, it doesn't work. But these are computers, and like humans, they do have memories. And indeed, computers have probably more reliable memories than me. I make it up as I go along, whereas these things kind of go, I'm going to tell you the service for .net don't forget it. [George: Yeah]. And by the way, here's a record that says, hang on to this information for, I don't know, seven days, [George: yep]. And so I faithfully go fine for the next seven days. Whenever I see a domain name ending in .net I won't ask the route. I'll go straight to the service for .net because I'm reusing that cached information. George Michaelson 17:59 Caching is a miracle, an absolute miracle of efficiency. Geoff Huston 18:04 Well, caching is a miracle. Let's just leave it at that. And caching is what makes the DNS work. And so, in essence, what the root zone actually sees are just the cache misses, the things that aren't already distributed out there. And as these special recursive resolvers go searching for answers. They only ask the root if they find a top level name that they don't already have a cached entry. George Michaelson 18:30 This is not an absolute statement, because clearly some number of queries to the root are people who have just switched on their machine after an outage or a recursive server has been rebooted by your ISP. What you're talking about here is statistically, in volume, the things seen at the root, statistically, overwhelmingly, 99.999% are going to wind up being the things that the answer: Nup that doesn't exist. Geoff Huston 18:56 Well, those are the ones that kind of make it to the root, because, in theory, all the other queries should be trapped by recursive resolvers in theory. Now I say in theory because there is, I should have to digress at this point. And let's talk, she talk about who serves this root zone, because, again, this is one of these anomalies from the 1980s that just some reason in 2025 you look at this and go, Wow, dude, the hippies are gone. [George: Yeah] you know, haight ashbury is over, dude. No, what's going on? Because there's this little little bubble, and it's a Time Warp of enlightened altruism. George Michaelson 19:36 We'll do this in everyone's wider interest. Back in a day when costs were kind of within the limit, and it felt like the right way to do some mutuality, Geoff Huston 19:47 and the Internet was free and so on. And when Jon Postel was looking for volunteers AKA suckers to do him a favor and run one of these root servers, it didn't come. With money or even a contract. It simply said, Can you do this for me? And if the answer was yes, it was kind of right, buddy, your horse. George Michaelson 20:06 But that that mutuality actually was an amazingly good choice for the time, because it was effective and it was done reasonably quickly. And these were the technologists who had actual interest in achieving an outcome. We're talking groups like the research networks in the military, NASA, the university sector, people all over the place for saying, You know what, I think we've got a piece of this. Let's pick up the piece and do something. Geoff Huston 20:34 Donate our efforts to the cause of the DNS and be a good net citizen and do a bit more than I have to to make all this work. Yes, it was great at the time. It's still in operation today, [George: right] It's still what keeps the DNS working today. [George: Yeah]. And while a whole bunch of folk are making a whole bunch of money out of a whole bunch of Internet, there's this little pool of 12 folk, 12 organizations, who are still running these copies of the root zone for the DNS for free. And you sit there and go, well, jee, thanks a lot. You know, love your work. I take it. It's just a machine or two. You know, a retired, old, retired, old Dell server out the back, small, right? [George: er... No], no. The root servers actually have got together over the last few decades and have organized a lot of things. And one of the things they organized was this root server Advisory Committee. We call it in ICANNese RSAC, and they do a lot of good data and they actually publish as part of this RSAC data set the number of queries per day seen by all of the root servers. There are 12 organizations that serve the root some of them have bigger sets of servers. Some of them have smaller but all are 12, and they all contribute to statistics. George Michaelson 21:53 For those of you who are about to send in an angry message saying, aren't there 13 name servers? Please be aware that one of the entities that runs the route runs two letters. Geoff Huston 22:05 That's right, let's, let's call them out, A and J are both operated by VeriSign, and we might even get into why, or if not, we'll hold it for another episode. But, you know, historical anomalies, etc, etc. So anyway, they publish their data, and yesterday, ish, there were, oh, 140 billion queries at the root zone. George Michaelson 22:28 That's a large number. That's a Carl Sagan number. That's billions and billions of queries. Geoff Huston 22:33 It's insane. It's insane. These are all cache misses, George Michaelson 22:37 these, these. This is not the number of DNS queries to run the Internet. These are the number of DNS queries that are hunting for the answer. No, doesn't work, [Geoff: right] So 130 billion. No, Geoff Huston 22:52 let's roll it back for two years. Two years ago now, the Internet is largely a saturated thing. It's not massively expanding. There's no huge tracks of unconnected people or unconnected things. George Michaelson 23:07 Well, there are, but they're unconnected in every possible sense of the word. By now, if you have any kind of phone or similar digital device, you are on the Internet. Geoff Huston 23:17 The Internet is growing closer to population growth than it is to this unnatural chasing unmet demand. And so the industry itself is seeing sort of five, 10% growth per year, if you're lucky, five is more like we're plateaued out two years ago. There weren't 130 billion queries a day. There were 90. George Michaelson 23:37 That's a lot more than 5% growth a year. Geoff Huston 23:40 That's right, it's a compound growth of 25% per year in queries to the root. Wow, when you actually plot this graph, and maybe we'll do some URLs at the end of this episode, who knows? As you can see it, but you know, it actually rose to a higher number, 100 and 80 trillion, billion, sorry, in January 21 and then all of a sudden, didn't quite have but plummeted. [George: Interesting]. The answer, oddly enough, was Google Chrome. Google Chrome. When it first booted up, I was sitting there going hum. I wonder where I am, and I wonder what's around me, and I wonder if the DNS environment that's operating around where I am is usable or not? [George: Yeah], I wonder if there are strange bits of DNS interception. So what it did is it tested it by doing a number of queries for so called chromoids. [George: Yeah]. These are random strings, I think, between 11 and 15 characters long, a single label that did not exist. And every time chrome started up, it did a couple of these queries, George Michaelson 24:48 yep, because if it's in a faked out Internet, there is a greater than zero chance someone's going to say, Oh, you look like you want to find something. I'll send you over here, whereas in the real Internet. Geoff Huston 25:00 Yeah, I will substitute a fake answer for this, non exist. George Michaelson 25:02 But in the real Internet, it should be Nup that does not exist. Geoff Huston 25:07 That seems like an innocuous thing. George Michaelson 25:09 Oh, okay, I guess there's this sting in the tail coming. So how many people use Chrome around the world?Geoff? Geoff Huston 25:16 Why? Current estimates? It's well, in the billions. Yes, well, in the opinions, Chrome has a market share of, you know, somewhere around 60 to 80% globally, it's a lot of things using Chrome a lot. And so when Chrome's one or two queries, gets played out in reality, 60, 60 billion queries a day, you know, oh my God. And so when Google were kind of informed that this was kind of kind of bad. They did turn it off, and they use a different technique now, and that's great, but the relief at the root zone was only temporary, and yet again, you find a small bunch of 12 doing this out of the goodness of their heart. No one's paying for it, answering 130 billion queries a day, and two years time, if this continues, it's going to be quarter of a trillion queries per day. There's no stopping it. George Michaelson 26:10 Having said, Well, it's not just the spare PC that we've stuck in a cupboard out the back. These people, in order to sync this much traffic, have to run somewhere around four to 5000 machines in a giant, any cost system worldwide. They have to put capital in the ground. They're spending money. Geoff Huston 26:29 We're going to get there. We're going to get there. But the first thing I want to touch is who's paying to have those queries answered. Because, you see, the Internet was this de regulated temple to capitalism that we were going to get rid of the stultifying hand of communications regulation and government monopolies and all that kind of stuff markets were going to run the Internet. And yes, it's quite normal to pay for web hosting. We all understand this. It's quite normal. Google notwithstanding, who even pay for other services? Not everything is free, funded by advertising, and that's certainly fit and proper. But who funds the DNS? George Michaelson 27:09 I haven't seen a DNS Bill arriving on my desk. I don't have a component of tax that says the DNS tax. Geoff Huston 27:16 Oddly enough, you're paying for a tiny bit of it, because if you're a customer of an ISP. You expect your ISP to actually operate a DNS recursive resolver, and that's all bundled into the price of access. So you get your recursive resolver and you're funding it, you're paying for as a customer, but when that resolver that you are partially funding makes a query to the root server, no tax, no. Check, no the bills in the mail, nothing. It's unfunded. The root servers get no money, and you kind of go but why is it different from any other authoritative server? If the DNS is in an economics free zone, why hasn't it collapsed years ago? George Michaelson 27:56 Good question. Geoff Huston 27:57 Well, all other authoritative servers are funded by the folk whose names they serve. So if I want to serve apnic.net I go to the market and say, Well, I have money. Who's going to do this? What's the kind of service you're offering? How much do I pay? And so all the other authoritative servers are actually funded by the folk whose names they host. It's a perfectly rational market. There are players. There's pricing, [George: yeah] it's competitive. It actually operates quite efficiently, but logically, who pays the route? George Michaelson 28:29 Yeah. So you could construct, I mean, this is George inventing things. You could construct a regime of an impost based on size of zone, amount of query, you could invent a way to say, here's a fair share. Geoff Huston 28:46 You could. George Michaelson 28:46 You could do that. Geoff Huston 28:47 You could. But let's go back to one of my opening comments. The DNS root servers are theoretically privy to a whole bunch of very, very valuable information. [George: Oh yeah], because if you're looking for a name that isn't in your local cases, you ask the root server, the full bananas, the full name, it's it's a huge privacy leak. And so a lot of folk who would say, Yes, I'll be a root server. I just want the ability to sell all the information I glean from DNS theories in real time. George Michaelson 29:18 I'd like to make profit out of this passing service that I will give to you for free. This invokes the classic Internet meme, if you're not paying for the service, you're not the customer, you're the product. Geoff Huston 29:31 We don't like that. We think that's a bad move, and so we say to these DNS root servers, if you can't monetize the traffic you're seeing, that's not part of the game. You can't sell to the captured top level domains your authoritative service. You can't do that either. I'm sorry. You've just got to keep on doing this for free. You've got to keep on doing this as an altruistic thing, because no one's going to pay you. But by the way. A the amount of money you spent last year isn't enough for next year. George Michaelson 30:03 The rate of growth here exceeds the rate of growth in the underlying Internet at scale as a mature system. Why is not really that important. The fact remains, this query load exists. Geoff Huston 30:16 So how do we grow? Which is really getting into the nub of this problem. How do you grow it? Well, you've got 12 operators and 13 slots, why don't you add another operator and have 13 operators? The answer is, Well, we already filled all 13 slots. Well, why not have 14? What's so magic about 13? [George: Ah], a long, long time ago, we decided that 13 was the maximum number of servers we could fit in an answer that managed in a V4 only world at 512, octets. [George: Yeah]. So if you had 14, it wouldn't fit. However, later experiments, and the experiment called Yeti was was an interesting case in point actually show that that's really a bit of a fiction. You can have 14, you can have 20, you probably have 100 and quite frankly, even if you don't list all the name servers in a response, it doesn't matter George Michaelson 31:11 if you sample through a pool of people, you could do this Geoff Huston 31:15 No recursive resolver. Is that patient anyway, if you will, six or even eight name servers don't respond. Most recursive resolvers give up their hands going, but, but there is no answer. I'm tuning out for a while. Go away. I'm just not going to do this. Nothing's that persistent. So even if you have 200 name servers, I really don't think there's a resolver out there that would patiently pull through all of them before say, Well, I'm sorry, that thing really doesn't exist. You know, this doesn't work like that. George Michaelson 31:44 I think we're smart enough as host and audience to get to the real point here. If it actually isn't a technology problem that's stopping us growing the pool of 13, it's politics, isn't it? Or one of the many words that means politics Geoff Huston 32:00 It's the intractable issue of going, how do you decide who gets crowned, who gets a cache of running a root server? Because, quite frankly, that's almost an insoluble problem. You're pitting willing sovereign governments, willing industry consortia. All kinds of folk seem to say that there's some kind of elevated status in running this, but there's a deep suspicion that if anyone thinks that this is a task that gives you an elevated status, maybe they're not the right person to do the job. George Michaelson 32:33 Yeah. So as long as it's only 13 aging hippies doing the job, or 12, even if they've turned into cash cows and are doing okay. It's kind of a position of neutrality we can live with. But if you open the door to saying there could be more state actors, commercial interests and people that other people are suspicious of enter the room, as does the question, who decides? Geoff Huston 32:58 Who decides? How do they decide? To be perfectly frank, that was the core of the work of the root server, governance working group, George Michaelson 33:06 yeah. What is the way out of this mess? Geoff Huston 33:09 How do you make these people accountable, and how do you figure out the rules for adding and removing now they're still working away years later. Bless their hearts, I'm still suffering. I don't actually, don't want to talk about that, because it's still work in progress. But what I want to, I suppose, come to the conclusion, is that augmenting this number 13 to even 14 presents some pretty deep and significant problems. And maybe there are other ways out of this mess. George Michaelson 33:38 Oh, I do like, is there another way? Geoff? That's a story that we can run Geoff. Is there another way? Geoff Huston 33:46 Well, the first thing it was a hard fought battle, and it's actually it parallels the entire adventure of computing that once you couldn't up the clock, you built two computers and strap them together and exploited parallelism. So if you're trying to run the entire root service from 13 aging single platforms, you're going to die. So what you actually do is you make it a lot more than that and put them out in parallel using this technique called anycast, where there's still only one v4 address and one v6 address out there. But depending on where you are in the world, the routing system will take you to the closest platform that responds to that address. So here am I in the Deep South Pacific, and there are root servers out there for various letters. I don't think they're in my town, but there's certainly 300 kilometers down the road in Sydney and in Melbourne and in Brisbane and in Perth. George Michaelson 34:43 So functionally, it localizes service and it cuts down the scale of the problem. All things being equal, if we put a whole sheaf of things to one side, the chances are you're not looking at dealing with 300 billion queries at a single point Geoff Huston 34:59 exactly. And quite right now, there was a census in March 2025, thank you. Rootservers.org and there were 1513 instances of a root service. And in each instance there might be more than one platform actually operating that service load balances whatever. And so there's not 13 or 12, there's 1000s, and that number continues to grow, right? So every single letter, in one way or another, is kind of solving the scaling problem by going towards any cast. It's been enthusiastically embraced, and to some extent, that's what keeps the service running today, George Michaelson 35:40 yes, but I've got to say in the back of my mind, the rate of growth you projected there 25% compounding year on year. That's still a capital nightmare. That's a lot more equipment, a lot more points of deployment, a lot more people running systems you're running to keep still. So in some ways, this is the technology solution that kind of works. And I've got to say, Geoff not having to grow the 13 means you shut the door on the political problem. It's okay, but man, this is rising cost. Geoff Huston 36:10 Well, you shut the door on new sources of money too, including Sovereign Money. So it's kind of Whoa. Did we do that? Yes. And the real question now is, how do I make it grow 25% bigger? Do I increase my any cast again and again and again? Where does this end? And the answer is, it ends when you go broke, son, you know? [George: Yeah] there's no other way out. Exponential growth is about the harshest master we know. I George Michaelson 36:38 think the idea we're heading to a grand crash in DNS is not a good idea. So foreseeably, this can't go on forever. Check, we just went to a technology. Is there another way out? Check, I'm actually thinking, let's do a bit of recursion. Is there another [Geoff: another way] Is there another way out. Geoff Huston 37:01 Well, okay, so the beauty of it is that the root server data is not entirely opaque, even though it's treated with a lot of caution, and it's in personal data in there that things are blurred out a lot. But one thing we do know and do track, and it's in this this data from RSAC, the root server advisory folks, is the amount of responses that say the name you asked for is not in a defined top level zone, ain't there, dude. And even today, 50% of all the answers proffered by these root zone servers are simply nup: NXdomain that is not a name. George Michaelson 37:38 50% so we've got something growing 25% year on year, and half of it is simply giving the answer no to the question, does this exist? Geoff Huston 37:50 When Chrome was running in full swing with the chromoids, that number got as high as 80% and when Chrome got rid of the chromoids, it started to fall and as more recursive resolvers tried an alternative here, this number actually did get lower. The problem is that there are infinitely more names that don't exist the ones that do. If you remember your pure maths, maybe you don't there George Michaelson 38:18 wasn't good at maths at school Geoff. That's why I'm not an engineer. Geoff Huston 38:23 There are infinitely many more real numbers than there are integers. And in fact, between two integers, there's an infinite number of real numbers. You know, the things that don't exist way way out populate the ones that do. And so conventional caching for negatives doesn't work, George Michaelson 38:40 right? But this feels like the opportunity staring us in the face. How about we cache the things that do exist and say categorically, that's your blooming lot. That's all that exists. Geoff Huston 38:54 This was the piece of brilliance in the thinking behind DNS security, security mechanisms for DNS, because not only did they want a way to say this is a real answer, believe me, they also wanted the way to say this is a "NO" answer. The name you're asking for does not exist, believe me. Now, their challenge was that they didn't think at the time, and as I said at the time, they couldn't create a cryptographic answer on the fly for every query. They just did not, at the time, have access to that much compute. So they wanted to pre compute every possible answer to every possible query without knowing what the query might be George Michaelson 39:35 With an infinity of queries, that sounds like a stretch Geoff, so you're saying they had to pre compute the ability to say no to an infinite number of things. Wow. Geoff Huston 39:47 Well, it actually isn't as hard as you think. Let's go back to my dictionary analogy. Now I have, I don't know, 60,000 odd words to find in English and their Thank you. Dictionary neatly ordered, and I develop a new . List, the list of character strings that aren't in the dictionary. Now you could say that's an infinite list Geoff, and you kind of go, yes, it is. But what I do is I define it by a set of ranges, [George: right] So between, I don't know, cap and cat, let's say there are no other no other words, between those two English words. I just simply say [George: Car, car]. But the way I can do it is it with a similar number of entries as the words are doing George Michaelson 40:33 caster sugar. Geoff Huston 40:36 I've started something that's getting really hard to stop here. George Michaelson 40:40 I'll stop, I promise. But I get the idea you'd say the two words that are in the dictionary, and now you only have to say the prefix. It parts of words that just don't exist. Geoff Huston 40:51 There are as many gaps as there are words, and so the entire universe of words that don't exist can actually be summarized by the same number of words that are in the dictionary, 60,000 or whatever they are, by actually saying, George Michaelson 41:06 because they're ranges, Geoff Huston 41:07 because they're ranges, and they summarize in each range all the words that didn't exist within that range. And then the cut through in DNS SEC was, let's do that for the DNS. Wow, brilliant. Because now, with just around 1400 negative ranges, I can actually have all of the "NO" answers pre computed. Ask me anything, dude, ask me anything. I can give you a no that's pre computed for the range of real words that span the actual word you asked for. George Michaelson 41:39 So we already have caching. I've never been online before. I come and ask the root the question for.net I acquire.net I now ask for dot nerd, and it immediately says back, no, that doesn't exist. And what's more here is the range between dot something else and. net nothing lies in that space. If I cache that, I will never ask another question in that range, because I know I've been told it doesn't exist. Geoff Huston 42:08 I don't need to ask the route. I have now stored a range, and that range is good for reuse for the lifetime of the cache. I had George Michaelson 42:16 to get that negative answer. I've got to receive that range answer saying, here is the range of things that don't exist I've got to Geoff Huston 42:24 load on the root servers. Is the same as the load for positive answers. Again. Think about it. There are 1400 real names and 1400 gaps. And if you uniformly give me a set of queries for real names, you know, the cache will start working if you uniformly spread the distribution of names that don't exist after approximately, I don't know, 1400 queries, at best, 3000 queries, probably at worst, I'll cache the entire range of the root zone, and I'll never ask the root zone again for the lifetime of The cache. George Michaelson 42:59 Caching is magic. Geoff Huston 43:00 Caching is magic. This negative caching, aggressive insect caching, RFC, 8198 been around for years and more to the point, been implemented. Bind 9.12, unbound, one point 7.0, knot resolve a two point 0.0, we're all doing it. We're all doing it now. Why is the negative response rate for the root at 50% we're all doing it. So we go back to some other data. We look at one of the larger recursive resolvers, where we actually have useful data at the recursive resolver level, a large public recursive resolver, the negative answers are around six to 7% but at the root zone, it's 50. George Michaelson 43:46 These don't add up. Geoff Huston 43:47 Nice, try do but there's something else not working here. Something's kind of going wrong. Okay, we'll try two approaches. Let's go for a third. Why not? George Michaelson 43:57 Oh, another, another, another. Geoff Huston 43:59 Anyone who's sort of lived in this space and actually had a look, and don't forget, once a year, OARC, the DNS operations research firm, get the root server operators to collect a day's worth of data and publish it, not all the private query names and so on. It's privacy kind of filter, but there's still enough information there to say who asks the root zone questions, who's asking, and you should find a bunch of recursive resolvers, George Michaelson 44:25 right? So use a bit of a population survey to understand the nature, not only of what questions get asked, but what IP addresses are seen asking those questions. Geoff Huston 44:35 And what you won't find is the most popular open recursive resolver on the planet. Google's public DNS service, 8.8.8.8. Its servers don't ask the root zone any questions at all, not even to do negative caching. No questions. George Michaelson 44:52 Well, I gotta say, Geoff, it's unquestionably working as a provider of DNS because I'm using it on a daily basis. And it does not fail to answer my queries. So at this point, if it's not asking upstairs, that sounds like magic. Geoff Huston 45:07 Unquestionably, in both senses of the word George, it's working. But equally, you know, it doesn't ask questions, and the answer is, well, why not? Well, it used a sort of a dimly used feature that I don't think was well understood at the time, but applies brilliantly to the root, which is a thing called AXFR in the DNS Hi server. If it's not against your policies, just give me your entire zone. Just give me everything. And I'll take it from here for a while, and a little later on, I'll ask again, and I'll take it from there. And we can do that for the root, and a number of folk do do that for the root. The beauty of it is it enhances privacy. I never get seen asking for anything again, just the entire root. George Michaelson 45:49 Once you have a copy of the route, you're never asking those questions coming through, except for maybe negative answer questions. But if you've implemented things, right? You don't even have to ask them, because fetching the root fetch those range answers to say, Nothing lies between here. Geoff Huston 46:09 I've got everything. There is nothing else that the root zone can answer. I've got the root zone. Now you go, Wow, that must be a big zone. Yes, 2.2 megabytes. George Michaelson 46:19 This is something that wouldn't work for .com there's a reason.com might say, No, I'm not giving you the whole zone. Geoff Huston 46:25 There are a whole bunch of reasons, including including commercial [George: Yeah], and I'm not suggesting that this is suitable for everyone, George Michaelson 46:32 But at the root, that's public interest. That's the space where we need to know this stuff. Geoff Huston 46:37 The route is public and quite frankly, if every recursive resolver on the planet grabbed a copy of the root zone and held it for a day and then did it again 24 hours later, you wouldn't have a root zone server problem. Because, in fact, the recursive resolvers, who are meant to be the intermediaries between end users and you know, the root zone, would actually say, I've got this. I've got this. I don't need to ask no root server folk, because I already have the answer. It's sitting here in the 2.2 megabytes of root zone data. I can do this just as easily and more quickly and way more privately. Why aren't we doing it? George Michaelson 47:15 Yeah, why isn't everybody doing it? Geoff Huston 47:19 I think there's a certain amount of suspicion over the use of AXFR and the DNS, and it's kind of who gets to serve the entirety of the root zone. And I would actually argue that the existing DNS operators are not the best folk to do it. We have, in the last 20 odd years, reconstructed the web, and we have reconstructed the web in a brilliant fashion using content distribution networks. We replicate the data like crazy and serve precise replicants of this data all over the planet using a whole variety of CDN platforms. So what instead of using the DNS, because now the root zone is just a blob of data. Put that data behind a URL, and it's deliberately it's an address URL, not a name URL. So I p1 dot 3.4, or something. George Michaelson 47:53 You've put two things forward. Well, actually, three, [Geoff: yeah], one, this AXFR thing makes some people uncomfortable, but what matters is the data. Two, if we're looking at a way to do it, we don't have to limit ourselves to the magic 13 who currently provide DNS because three, it's just the file. It's just the stream of data. You can fetch it using any mechanism, and the mechanism that seems to scale that we're depending on globally is cached content distributed widely in the web, [Geoff: right] So you're saying fetch this data from the web, Geoff Huston 48:50 because I would actually argue the root servers are actually constructed to do fast UDP at bulk. They haven't been built and engineered to deliver whole zone copies, even if it is only 2.2 megs over TCP in bulk, that's not their prime speciality. When you kind of go, well, that's a bummer. And the answer is no, it's not, because there's a whole web industry out there just over your shoulder that does that in its sleep and servicing lets see.. around 12 million recursive resolvers with a copy of the root zone every 24 hours is a blip that almost no CDN would even notice in terms of the traffic requirement. George Michaelson 49:30 Why should I trust this file if I get it from one of them? Couldn't they be mucking around with the state of this zone? Couldn't they be changing things in it? Geoff Huston 49:37 So of course, they could. George Michaelson 49:38 Where's the trust? Geoff Huston 49:39 Do you remember the other piece of hell that I was engaged with the rzerk folk, the root zone evolution and this neat little record called zone message digest, zone MD, as long as you have a current copy of the root zone key, signing key, which only changes, well, I was going to say. Every five years, but less frequent than that. It's a pretty static key. You can check that the copy of the root zone you just received is the genuine article, so you have a way of actually determining whether it's authentic or not. And so the only other question is, How often should I check to see if it's changing? And the answer really is, it's the root zone dude, once a day is fine. George Michaelson 50:23 That runs counter to my intuitive sense. I got the feeling DNS was a lot more dynamic, and we had to be prepared for more frequent change. You're saying actually, at this particular point, we don't have a cycle of constant change here. It's infrequent. Geoff Huston 50:39 If you want caching to work, then you've got to sort of allow some slop in time synchronicity, because you want the local cache to be as effective as they can be for the longer period as you can that's what make caching work. If you want a system to be highly responsive and change rapidly, caching is working against you, and quite frankly, in the DNS with the way these cache timers work, it's a 24 hour cycle in reality. So why don't you simply say, grab a copy of the root zone, use it for a day, exactly 24 hours, and then get another copy and say to the dear folk at IANA, you can make as many changes as you like, but only once every 24 hours. So you can change the root zone and 24 hour cycle, be precise, do it at the same time every day. That allows everyone else who is pulling a copy from their local CDN to do so in a 24 hour cycle, and you're basically all current. This kind of puts the burden serving this data on the folk who serve this data as a profession really efficiently, CDNs are good at this. I'm not asking the CDNs to do DNS, not at all. This is just a web object. You could have the root zone encoded in JSON, George Michaelson 51:54 and it's about two meg of compressed data. It's actually a small chunk that would be smaller than the photographic images they ship routinely every single day to all of us in newspapers and magazines and Reddit or what. Geoff Huston 52:09 t's not the intractable problem that the same piece of data is presented to the UDP based root name servers. So instead of doing 140 billion queries a day and growing and trying to answer things within milliseconds all the time. Everywhere you simply go, fetch this file, use it for 24 hours. When that elapses, try again, and you tell every single CDN on the planet, no naming rights, no nothing. You're doing this as a service to the rest of us. You hardly even notice the traffic. Why don't you also serve the root? So what do you need? A single IP, any cast address, maybe two, one, in Vsix, one, in Vfour. And you go, well, a bit like AS112, another interesting conversation about how to do this. A bit like AS112, you don't have to apply to anyone. You just have to serve it. And everyone knows that you're serving the right thing. If the zone MD message digest actually gives you, yes, that's authentic. It's the real deal. George Michaelson 53:09 It's a checkable proposition. Geoff Huston 53:11 Exactly, why are we doing it? George Michaelson 53:13 What's stopping us from doing it? What's stopping us? Geoff Huston 53:16 Well, interestingly, a tiny delta. And you, dear listener and me both have the ability to actually change this. This work is actually configured in most DNS recursive resolvers, root zone mirror, some have made it even a one liner, like in bind and unbound and power DNS. You know, it's there. What it isn't is the default when you bring up a new copy of bind or unbound, or whatever your favorite recursive resolver is, if it by default said, unless you tell me something different, I'm going to snarf a copy of the root zone from this well known, any cast, you know, URL, and I'm never going to bother the root zone servers Ever again if you want that from your favorite DNS recursive resolver vendor, then ask them you want in inbuilt default behavior of doing a local root zone cache. George Michaelson 54:13 I kind of feel like I want it on two grounds. I want it on the community benefit ground, reduce burden on a central function and avoid growth, and I want it on a privacy ground. I don't see why I should be telling everybody promiscuously where I'm coming in the DNS. I kind of feel like privacy first, this is something to think about doing. Geoff Huston 54:34 I can give you a third reason. Geoff gets his life back, yay. All of a sudden, George Michaelson 54:42 you did commit very many sins. Geoff Huston 54:45 I'm sure there are more ways to pay penance. But now the point is, in many ways, this is something that's staring us in the face. We have totally transformed the content industry by, if you will, running and going where our nose leads. Is to replicate content, to do scaling in different ways, but the root zone is kind of stuck in this anachronistic version of this little bubble of altruism, and it's just almost impossible to leverage them out of it. And I'm saying, fine, not your problem. Recursive resolvers are actually the key to solving this. You change your default behavior. We can radically reduce the amount of traffic hitting the root zones incremental DNS queries, and simply push all this back to where there's more money, more economic sense and more silicon into the recursive resolvers, and get them to actually locally serve the root zone and simply not ask the root zone servers incremental queries ever. And you go, well, where's your authority from? Well, it comes from IANA, and they have a bunch of relationships with the content distribution networks, the same as any other web publisher. George Michaelson 55:50 It's kind of taken a tension around the magic 13, the 12 parties, which party should come into this, and moved it into a different domain. There is no exclusive here. We can have any number of people doing this CDN, and it's a contracted relationship of community benefit. It's not a control point, and it's not a monetization point. It's actually the anti set. It's a do a thing to avoid having any of those risks. Geoff Huston 56:19 Absolutely, and I suppose we're putting a lot of faith in cryptography to get it right. That's true. You know, you're getting a fake if zone MD does not compute, as simple as that. So there you go. George Michaelson 56:29 Interesting outcome, very interesting outcome, Geoff, Geoff Huston 56:32 In a couple of weeks, we'll solve some more of the universe's problems. George Michaelson 56:37 Geoff, if only life was so simple, but that's been fascinating. Thank you, and you're right. We'll put pointers to articles and RFCs about this online so people can understand how to look at this for themselves. Geoff Huston 56:50 Absolutely. Thank you, dear listener, thank you, George. See you next time George Michaelson 56:55 If you've got a story or research to share here on ping, why not get in contact by email to ping@apnic.net or via the APNIC social media channels. Also remember that the measurement@apnic.net mailing lists on orbit is there to discuss and share relevant collaborative opportunities, grants and funding opportunities, jobs and graduate placings, or to seek feedback from the community on your own Measurement Project, be sure to check out the APNIC website for All your resource and community needs until next time you