Willem Toorop 0:00 Yes, so our approach was to first do a survey with all the root server operators and ask them about what software do you use to serve the roots and what operating system do you use, and how do you configure your operating system, and how do you configure the software, so not all of this is public, but much of it is public, so I can say that a large part of the root is served by open source software. George Michaelson 0:40 You're listening to Ping, a podcast by APNIC, discussing all things related to measuring the internet. I'm your host, George Michaelson. This time on Ping, I'm talking to Willem Toorop from NL Net Labs. Willem has been on Ping before, talking about kernel level packet filtering methods, but this time he's exploring a measurement activity about the names of the DNS root servers. Root name servers are part of the global DNS system, and they provide basic bootstrapping for every other DNS resolver and authoritative name server worldwide. They also handle query load relating to the global DNS system all day and every day, there are 13 distinctly named root servers, each one identified by a letter from A to M. There are, in fact, 1000s of machines providing this service behind each letter. The letters are named under a fully qualified domain name, a special zone, root servers.net This was instantiated in 1995 It's delegated as normal under the.net zone, which in turn delegates from the root dot zone. So there's a kind of circular dependency risk that to find a root server you appear to need to know about the delegation of.net to find root servers.net to find the given server. This is resolved by the contents of an initial priming response. This response provides the additional glue information to seed direct knowledge of how to find each of these named root servers as is, without all the intermediate logic of the circular dependency, something that has to be just understood for now is that this glue is inherently not signed in DNSSEC. It's insecure. The priming response is very carefully curated for both its size and the content. A document published in 2017 by the ICANN Root Server System Advisory Committee, or RSSAC, explored the impact of applying DNSSEC over this domain, which in turn would add DNSSEC signatures to the priming response and impact a goal of keeping this a small 512 byte backwards compatible form. This would continue to work on all legacy systems worldwide, so changing the naming model of the roots has impact on something we want to protect. Nothing happens quickly in DNS evolution, and this naming question has now been under consideration for over a decade. Willem and his colleagues at NL Net Labs produced two reports with collaborators at SIDN Labs, also located in the Netherlands. The reports reflect on the candidate naming schemes, application of DNSSEC, and the impact on the bootstrap fetch. Willem, welcome back to Ping. Willem Toorop 3:38 Thank you, George. George Michaelson 3:39 So people may not know that Willem has been on ping before, talking about some work that he did, putting packet filtering and processing logic into the Linux kernel. This time we're going to be talking a little higher up the food chain, a little bit more in sort of layer nine space around decisions of what we do in the DNS, aren't we? Willem Toorop 4:03 Yes, and at the same time it's also very lower level because it's about the roots. George Michaelson 4:09 So it has a little bit of both the structural behaviors of the DNS and this question, how do we proceed doing things we do in the DNS? So to lay the tracks a little. Could you tell us a little bit about why you're doing this report? Who asked for you to do this analysis? Willem Toorop 4:28 Yes, so. This is about indeed two reports I did as part of a consortium together with SIDN Labs, so there were a few people from NLNet Labs and a few people from SIDN Labs, so me, Benno, and Jogos from NLNet Labs, and Moritz and Marco from SIDN Labs, and this is in response to community requests, so there's the Root Server System Advisory Committee, the RSSAC. And they wrote a report 28 which is about different naming schemes for the root, George Michaelson 5:07 yeah, Willem Toorop 5:07 to assess if the root, as it is currently designed, has the most optimal or the best naming scheme. George Michaelson 5:15 So that is an ICANN grouped committee, that's a review body that kind of discusses aspects of the root service system as a whole, but you said there were two. Willem Toorop 5:26 Yes, the other committee is the Root Zone Evolution Review Committee, and they review potential changes to the root and can also do suggestions, for example, that Zone MD is now part of the roots that came from, yeah, George Michaelson 5:43 And they are also ICANN driven, but there is quite a range of participation in this. RSSAC is quite a large inclusive body. There's a large number of people. The RZERC is a little smaller, perhaps. Willem Toorop 5:59 Yeah, yeah, I'm not sure how many people, but it's all representative from different organizations, George Michaelson 6:07 so we might need to follow a different path in laying the tracks. When we talk about the DNS, there's the part which is to do with what is actually said, that is about contents of zones, things like Zone MD, [Willem: yeah] and there's the part which is who is saying it Willem Toorop 6:26 yes, George Michaelson 6:27 and the work you're doing I feel is kind of in a weird intersection, because in order to say who is saying it, Willem Toorop 6:35 yeah, George Michaelson 6:35 you have to actually put some data into the DNS, so it's kind of self-reflecting that this list of entities that tell you things, the name servers, the NS records, have to be in zones, and we are right up at the apex of this system, the imaginary dot that is on the end of every single name, there has to be a list of things that serve them, and they are called the root servers. Willem Toorop 7:03 Yes, George Michaelson 7:03 every root server has a name. Willem Toorop 7:06 Yeah, well, those are called the root server identifiers. Okay, little m, George Michaelson 7:13 right? And those names have to exist in the global DNS. Willem Toorop 7:17 Yeah, George Michaelson 7:18 and there's this operation that we call the priming function, where you try to determine how many of these there are, and where they are, and how you could communicate with them, which demands that you receive information about a zone and the structure of those names. Now, this study is about aspects of the structure of those names, isn't it? Willem Toorop 7:40 It's indeed about bootstrapping the whole DNS. George Michaelson 7:44 What was the primary concern that led people to wanting to ask some questions? Willem Toorop 7:51 I think the RSSAC 28 report, which is about different naming schemes, is just part of the job of this boot service system advised me committed to periodically revisit if everything is just as the best current state of practice, so to say, right. So they need to re-evaluate if there could have been a better naming scheme for the root servers. George Michaelson 8:19 So this naming scheme, it kind of has a consequence of the shape of the smallest packet you can send in the UDP DNS, doesn't it? There's a boundary that says if you try to encode this priming data in more than the length of this base packet, you've incurred protocol costs that has to cope with bridging into multiple packets. Willem Toorop 8:41 Yes, yes. So the size of the priming query is the motivation for the current naming scheme, which came from Paul Vixie. George Michaelson 8:52 Yeah, he did basic work quite a few decades ago now, Willem Toorop 8:57 in 94 I believe. George Michaelson 8:59 Yeah, and the behavior here again. This is something that we don't often talk about in the DNS. When you look at a packet that has a DNS request with a fully qualified domain name, you don't necessarily see the literal script string of ASCII characters that are the name. This thing called label compression is done on the name, isn't it? Willem Toorop 9:21 Yes. George Michaelson 9:22 What does that really look like in practice? Willem Toorop 9:25 In practice, because all the root name servers now have a very similar name in the current naming scheme, it's they have the identifier, which is a letter from A to M, [George: yeah]. So there are 13 identifiers, George Michaelson 9:40 yeah. Willem Toorop 9:41 And then there's a.to and the remainder is root "dash" servers.net George Michaelson 9:50 So, if we consider each of these names as instances of labels in a domain system, they have a unique component, and there are 13 unique values, They then share two common components, root dash servers as an intermediate and.net and under label compression each of these things has to appear once, but then the subsequent reuses can say just like that one, it's the classic compression technique of only sending the minimum amount and sending a linkage to say that one Willem Toorop 10:23 indeed, so the long part is compressed into two bytes, George Michaelson 10:27 two bytes. Willem Toorop 10:29 Yeah, George Michaelson 10:29 then having 13 of these means that you have acquired a cost of 13 of the unique 13 pointers back to the long string and 13 back to the other shorter string, Willem Toorop 10:41 because the glue, the address records are also in there, so George Michaelson 10:47 Ah... glue, Willem Toorop 10:48 yeah, glue, George Michaelson 10:49 this wonderful thing that you're told a label, but you're then given hints about what the label is, because otherwise you're coming back and asking a question again, and in priming you don't know who to ask yet. Willem Toorop 11:02 Yes, George Michaelson 11:03 so glue is additional data, and it's about addresses, Willem Toorop 11:07 indeed, indeed. So, instead of having 13 times three, the root-servers.net only have root-servers.net once. Yes, and all the other three times 13 minus one instances, just refer back to that, George Michaelson 11:23 and then additionally, glue is added. Yeah, and this glue has to be for both IPV and IPV, doesn't it? So this then grows the packet, and you have all the other DNS protocol components, and we arrive at the boundary size, that is, it's somewhere around 512 570 bytes, there's some magic number quality, it's just the state of the world. Willem Toorop 11:48 Yeah, George Michaelson 11:48 And Paul designed this naming scheme to get the most possible benefit that he could, but this was designed before DNSSEC. Willem Toorop 11:59 Yes, yes, George Michaelson 12:00 and this was designed before IPv6 Willem Toorop 12:03 indeed, George Michaelson 12:04 and these names aren't necessarily either in a sub zone under the route or necessarily directly in the route. There are decisions to be made. Willem Toorop 12:19 Yeah, yeah, yeah. So, in the current naming scheme it's indeed below the .net zone, so there's a strange circular dependency, especially if you think about DNSSEC George Michaelson 12:32 validating the trust in the path where you are told to go and ask somewhere else. So this inquiry is kind of the statutory need. Let's check if we're doing this right. Willem Toorop 12:43 Indeed George Michaelson 12:43 And it's an opportunity for you to review questions like, How does this work under DNSSEC? What would it look like if it changed? Willem Toorop 12:51 Indeed George Michaelson 12:52 This was the nature of the work you were doing. Willem Toorop 12:54 Indeed, so the original RSEC 28 report proposed six different naming schemes, George Michaelson 13:01 six? three would have been enough! Willem Toorop 13:05 Yeah, and one of the recommendations in this report was to, or was basically to request a follow-up study to see how resolvers would respond to those naming schemes, and that's what we've done. George Michaelson 13:20 So, can you give us a brief flavor of the differences between the six schemes? Willem Toorop 13:24 Yes, so it's actually five new schemes, because the first scheme is the current scheme. George Michaelson 13:30 Yep, and there's always that option, do nothing, so just leave alone. Let's measure what we've got. Willem Toorop 13:35 A very good option, actually George Michaelson 13:38 good option, Willem Toorop 13:38 a good option. The second scheme is the current scheme, but then DNSSEC signed. George Michaelson 13:44 Yeah, Willem Toorop 13:44 so you have to know that in the current scheme, the NS resource record set is already DNS second signed, right? The response to the primary query, George Michaelson 13:57 yeah, Willem Toorop 13:57 but root-servers.net that whole zone is not DNSSEC signed, George Michaelson 14:01 so there is this thing that you would call loss of secure entry point, SEP. There's no SEP over that namespace, which means you can't currently perform DNSSEC validation. Willem Toorop 14:14 That is right. So, the proposal, indeed, for the addresses of the root-servers.net this is not possible. There's no DNS acceleration of those addresses, right? George Michaelson 14:24 And so a change here would necessitate either signing over the zone or moving them into a zone that is already signed. Yeah, Willem Toorop 14:31 all the naming scheme proposals for root servers are all DNS check signed, by the way, George Michaelson 14:36 right? So all of the other five alternates were signed, [Willem: yeah], but they achieved the outcome with different structural forms. [Willem: Yes] some were bigger, some were smaller. Willem Toorop 14:45 Yes, definitely. Yeah, the different naming schemes have have different impacts on the size of the priming query, but it's also affected by the authoritative name service that serves them and also by. I, the query parameters of the resolvers, so yeah, but maybe I'll go through the list of the schemes. So the second one is current scheme, but DNSSEC is not signed. The third one is just having all the names within the root zone itself, yeah. So there's no delegation, George Michaelson 15:19 yeah, Willem Toorop 15:19 in between, George Michaelson 15:20 which would permit signing, because the root zone is signed, Willem Toorop 15:24 absolutely. Yes, and also, which would automatically mean that it would be signed, but maintaining the root server identifier, so each identifier has its own name, [George: yeah], so either just a single letter, [George: yeah] for the addresser or single letter dot root service. George Michaelson 15:43 So something which needs to be clear here is that when you look at a fully qualified domain name, these dots do not necessarily mean is a different zone. You can embed a name in a zone with a "dot" in it, and so these things are like potential zone boundaries, they're not necessarily a real delegation point, and one of the proposals was to embed strings in the root zone that included the dot, Willem Toorop 16:08 yeah, and then all root name server addresses would be authoritative within the root zone, and so automatically would give signatures, so the fourth proposal is a shared delegated top level domain root-servers., so basically just cutting out the.net at the end, yeah, so root surface, they would, the root surface would get their own top level domain, George Michaelson 16:36 I might come back to that, but keep going, Willem Toorop 16:38 then the fifth one is each root server identifier, each his or her own TLD, so just the letter A would delegate to verisign, which is the root server operator that serves, George Michaelson 16:55 so instead of having a unifying quality, which has delivered in the label compression, it would explicitly recognize the independence of the entities providing it, and in some senses that is a different kind of resilience, isn't it? It prevents single point failure in naming because other parties are using other name forms. So, although I'm sort of thinking that sounds more costly, it had the potential to be more resilient. Willem Toorop 17:21 Okay, is it okay if I jump ahead already to the conclusion a little bit already? George Michaelson 17:27 Yeah, okay. I mean, people might cut out listening, that's okay. Willem Toorop 17:32 So this one, the fourth, [George: yeah] a TLD or a 2LD label, but a delegation for each root server identifier is the single scheme that did not have any problems with any of the resolvers. George Michaelson 17:47 Wow! that is very interesting. Willem Toorop 17:49 This was very surprising. George Michaelson 17:51 So, you, you have one more to catalog, either one or two more to catalog. Willem Toorop 17:55 Yeah, the last one is a single shared label for all the operators, so just one name, yeah, root servers, and then that one name has all the other assets, right? [George: Yeah] so now the root server identifiers don't have their.. George Michaelson 18:09 I remember this being raised at one of the Dallas IETFs as a potential model for naming, and it kind of invokes something in any cast that the proper number of things in a world of anycast is possibly one, Willem Toorop 18:24 yeah. George Michaelson 18:25 And this goes directly to that. Why do we have to identify uniquely, and one more after that? Willem Toorop 18:31 No, that's the sixth one, but that's if I can skip ahead again, because this one has also a surprising, but yeah, outcome, or [George: yeah], is that there's a variant on that, that if that single label for all the operators would be just the dots, like no label, George Michaelson 18:53 yes, Willem Toorop 18:53 then that's the only one which can guarantee to the resolver that all the glue in the additional section is authoritative. George Michaelson 19:04 Oh, wow, [Willem: yeah]. Choices, how did you go about testing these six propositions? What was your basis of measurement that would give you information? Willem Toorop 19:15 Yes, so our approach was to first do a survey with all the root server operators and ask them about what software do you use to serve the root and what operating system do you use and how do you configure your operating system and how do you configure the software, so not all of this is public, George Michaelson 19:37 no, Willem Toorop 19:38 but much of it is public, so I can say that a large part of the root is served by open source software. George Michaelson 19:46 Yeah, which in itself is highly beneficial, and something I personally enjoy. I think this is creditable. Willem Toorop 19:53 Yeah, yeah, indeed. So, for the open source authoritative software, Bind NSD. KNOT DNS is popular, so to say, or that, George Michaelson 20:04 so we have a diverse community of platforms and of software providing the prime protocol function. We are less exposed to risks of single source than we might think in this space, Willem Toorop 20:16 and also very important diversity of organizations that serve the roots, George Michaelson 20:22 yeah, Willem Toorop 20:22 which came apparent during this conference, actually, because I noticed something, something very odd. There's a little bit of sidetrack that I think is worth mentioning, that G Root and H Roots, which are the two US military ones, George Michaelson 20:40 yes, Willem Toorop 20:41 we're not reachable from the George Michaelson 20:43 so because we are conducting this interview in the IETF meeting being held in Shenzhen, Willem Toorop 20:49 yeah, George Michaelson 20:50 we are necessarily behind the boundaries of state control, which affect visibility of services, Willem Toorop 20:56 yeah, George Michaelson 20:56 and without objectifying either side in this because there are two parties in the delivery of service. It's just an observation. The two entities that have that vesting of authority are not visible in this community. It could be a domestic matter here, it could be a matter for the providers in the home economy. It's just the state we see. Willem Toorop 21:18 that's right. It's what was also far, it's not. It's very hard to detect which George Michaelson 21:23 I don't particularly feel we need to talk about that here, but it's an interesting observation. Willem, Willem Toorop 21:31 yeah, definitely. So it's good that root is served by first. George Michaelson 21:34 So you did a survey of the operators to establish the basis of delivery. Was this so that you could model behavior in a lab, or was it so you could understand when you perform tests in the real space? How did this then progress? Willem Toorop 21:48 Yeah, so all the tests were performed in a lab environment, George Michaelson 21:53 right. Willem Toorop 21:53 And software for that was originally designed by Paul Hoffman, actually George Michaelson 22:00 another member of our community who has done much work in this space. Willem Toorop 22:05 Yeah, indeed it's the Resolver testbed, so you can find it. [George: Yeah], ICANN git repository. We well built extensively, or we made a huge contribution, I think. Like, George Michaelson 22:20 yeah, Willem Toorop 22:21 yeah, George Michaelson 22:21 there's been other deliveries from the Dutch DNS research community in this space. SIDN have done work building DNSSEC test beds to provide visible tests against crypto architectures and algorithms, and I think it's the kind of space that bodies like NLNet Labs and SIDN are absolutely beautifully designed to address, take sources from other people, build and invest, make something, and use it to deliver a public outcome. It's lovely. Willem Toorop 22:49 Thank you. George Michaelson 22:50 It's really lovely. So you went out, you constructed a lab, you modeled the platforms you saw visibly providing root service, and then you took these six name forms and you applied them to this environment, Willem Toorop 23:05 yeah. Maybe it's also worth mentioning that there are also two commercial authoritative name servers used by the boot, which is the Atlas name server used by VeriSign that serve A and J, and Cloudflare, George Michaelson 23:21 yes, Willem Toorop 23:21 which is serving F, George Michaelson 23:23 yes, Willem Toorop 23:24 and so what we needed for our tests is how this software would respond to the priming query, [George: yes], for the different prime, George Michaelson 23:32 but it would be difficult to bring that into a lab. Did you enter into an NDA that gave you exposure to their software? Willem Toorop 23:39 No, but we managed to be with someone that had access to the Atlas platform and someone that had access to the Cloudshare platform for them to do priming queries for all the different name schemes with all the different resolver parameters, George Michaelson 23:57 so some component of your results were from a test bed and some were from live application with a partnership that allowed comparable results, Willem Toorop 24:05 indeed. And then we created simulating name servers within the test bed that would simulate George Michaelson 24:14 a clean room implementation of their behaviors to match the observations that were seen in the real world, that is very interesting, Willem Toorop 24:23 with a bit of an on-made lab software called LDNS testNS, George Michaelson 24:28 right, Willem Toorop 24:29 which is able to, on a very low level, simulate responses to certain queries. George Michaelson 24:35 So, in the real world, a system like Cloudflares or like Verisigns is handling millions of queries per unit time, it is absolutely saturated with load, and it exists in an anycast cloud. In a lab environment, you're not actually testing necessarily what happens when a million people ask, you're asking the fate of a single bootstrap, what is its transit through the protocol state, so. You only needed to respect the protocol behavior. You didn't need to service 20 million queries. Willem Toorop 25:06 That's right. We needed to test the diversity of responses, and not, yeah, if the identifier filers would be able to handle the load of queries. George Michaelson 25:19 How long did this take to run. Willem Toorop 25:21 Oh, this was a big effort, really, because, yeah, there are so many different parameters that we tested. George Michaelson 25:29 Well, there were the six different naming schemes, but then there are qualities within the priming behavior, each one of which had to be measured and assessed. So it's a large, multi-dimensional matrix of data. Willem Toorop 25:40 Yes, yeah, there was a lot of visual processing going on. Yeah, if you really want to know the extent of the amount of data that we processed reading the actual report, George Michaelson 25:55 yes, maybe put a conclusion in our blog that goes with the podcast, pointing at this, so you've given us two reveals of the final outcome. [George: Yeah], but I'm interested. Did you find any fatal one-way streets in some of the candidate name models in the software platform that you are testing? Willem Toorop 26:16 Yes, we did. But first, I like to point out, just because I think it's important that for this study it was important to look at the open source resource software, because the assumption is that those popular public resolvers or large resolvers, if there's anything problematic with a certain naming scheme, it's possible to reach out to them and ask them to change their behavior. George Michaelson 26:44 Yeah, Willem Toorop 26:45 but with all the different ISPs on the internet that use the open source resource software, this is not possible No for, George Michaelson 26:45 and so the people who ship that software, which includes NLNet, because you are now responsible for Unbound, which has a large surface of visibility globally, you have, in some sense, acquired an obligation, a burden in this. If you found a problem, you would be involved in writing mitigating behaviors. Willem Toorop 27:12 That is true, but the assumption is also that a large part of the internet will just keep running old versions forever, forever. So we also tested really old versions of bind knot resolver powerdns George Michaelson 27:28 bind back to bind 4 Willem Toorop 27:30 and unbound. We didn't get this to work, but George Michaelson 27:35 yeah, that's a little too old, even it doesn't do DNSSEC, that wouldn't be there. Willem Toorop 27:39 Yeah, George Michaelson 27:40 so what was the nature of a test? What would the typical test look like? Willem Toorop 27:44 I think the typical test would be just two test queries. Yeah, I'm certain we did this, and to start up the resource, so to say, because they also, those resource offers have different ways they do priming, and we were primarily interested in the priming query, because that involves the names of the root servers, [George: yeah], and a second query to see if everything succeeded, and what the current state, George Michaelson 28:14 so if the resolvers had acquired state, and you could demonstrate that state was functionally informing subsequent behavior. Willem Toorop 28:22 Yes, George Michaelson 28:23 so DNS, the protocol, I mean, it's beautifully simple. It's UDP for most people, send query, get response, but it has parameters. There are qualities inside the DNS packet that inform subsequent behavior, like please validate, turn on sending me associated data, only give me authoritative answers. Tell me if I have to move up to TCP. Tell me if you want to do a secure identity exchange, so that I can't do DOS attacks or fake packets. Were you testing those parameters as well? Six naming schemes, yes, four or five open source and two modeled private implementations resolver versions back to the dinosaurs, and all these parameters. This is a big, multi-dimensional test. Willem Toorop 29:12 Yes, it was very multi-dimensional, leading to huge lots of data to process, but it was sort of limited in the sense that we looked first, perhaps, to look at the different ways of query parameters that resolvers use. We looked at all the queries of data and inventorized all the different queries George Michaelson 29:37 so this is taking a large public data set that represents the surface of query seen typically at the roots, but in other nodal parts of DNS as well, and using that to inform the nature of queries that you present to this system, having been primed in this behavior. People may not realize that there's a subsequent behavior after priming where if. You like resolvers, taste test how quickly they're serviced by their back, and so different qualities in the behavior of the root system alter the dynamics of choices made in the resolver space. Applying real query load actually helps model, but what's the consequence in the longer term? It's a nice way of testing things. Willem Toorop 30:20 Yeah, George Michaelson 30:20 did you do a random sample from that? Were you taking like a one in 10,000 packet sample or Willem Toorop 30:26 no? We just took the whole of the digital data, which is so it's George Michaelson 30:31 the whole collection. Willem Toorop 30:35 Yes, yes, but this was just to inventorize the different query parameters, George Michaelson 30:41 right? Willem Toorop 30:42 Because what we noticed George Michaelson 30:43 so you sampled from that the selection of parameter settings, so that you would normalize. If you see this set, you tend not to see that set. Let's query this without that, that's the normal profile, Willem Toorop 30:56 indeed. And by doing that, we discovered that using all those different query parameters, which is millions, the authoritative responses to priming could be divided into eight different ways or in eight different groups, and within each group I think four different ways, depending on four different groups of query parameters. George Michaelson 31:23 so you managed to bring the number in this giant phase space of properties down to a more ordered subset of data. Yes, I think people are going to want to read the report to understand if we stay at the headline level, which of these six naming models, did you effectively knock out? Were there any? You said this just is too risky. Willem Toorop 31:45 There was one, which unfortunately is the top level domain shared with so cutting out.net basically top level domain shared... George Michaelson 31:55 just introduced too many behavior changes. It had higher risk overall, Willem Toorop 32:00 stopped some of the resolvers of doing resolution at all, which is a showstopper for the whole naming scheme, because those resolvers will be on the internet forever, right? So, George Michaelson 32:12 so there's a clear we can't do that. Willem Toorop 32:15 Yeah, George Michaelson 32:15 maintaining the current naming form you presumably demonstrated has no particular downside. Willem Toorop 32:22 Well, that was also surprising. Indeed, the current naming scheme also has issues, George Michaelson 32:27 really. Willem Toorop 32:28 Yes, and thanks to our reports. Yeah, so Duane has this ongoing, or has this topic in which he presented George Michaelson 32:38 Duane Wessels Willem Toorop 32:39 Duane Wessels which is also the root zone maintainer. Yes, very fine. Yes, so he's responsible for maintaining the root zone. Notice that, so in the past J root has changed its addresses, and there are still many, many resolvers that send query to the old addresses, George Michaelson 33:00 yes, Willem Toorop 33:01 that have not picked up the new address. Yes, from Primary. George Michaelson 33:04 I had a conversation with Dave Plonka, who was formerly at the University of Wisconsin, and he did a wonderful study of a DDoS against NTP servers many years ago. Now, at least two decades ago, he recently revisited. They're still seeing 20,000 unique addresses bombing that address family for NTP 20 years later, so it's not just DNS that has a long tail of this kind of behavior. Willem Toorop 33:32 Yeah, George Michaelson 33:33 some things are forever. Willem Toorop 33:34 Yes, but yeah, so for Duane this was a mystery until we wrote this report because we identified the software that fails priming or fails to learn the new addresses, then the new naming scheme it learns through priming is not altered, so it sees the names are have stayed the same, and then it wrongly assumes the addresses will be the same as well, George Michaelson 34:01 and were you to rename into one of the new schemes, would this heal itself? Willem Toorop 34:07 Yes, absolutely. George Michaelson 34:08 So we now actually have from a measurement activity a strong signal of benefit for deliberately making a choice to a new form of naming. Willem Toorop 34:18 Duane also tested this more severely, and he presented this during the last DNS OARC. George Michaelson 34:25 We might want to, we might want to look at that as well in the blog. That sounds like an interesting outcome. So, did you arrive at a recommendation? Willem Toorop 34:34 We wouldn't go as far as to make recommendations from this report, but we had one naming scheme that did not have issues, [George: yeah] which is each identifier getting its own delegation from the root, George Michaelson 34:48 yes, Willem Toorop 34:49 but the other schemes, so there's one that is really problematic, which is a single delegation for all the identifier, George Michaelson 34:56 which surprises me, I had expected this to be an. Naturally efficient and functional form, but the outcome Willem Toorop 35:03 Is it's very surprising that you know we are really, really certain, because we tested over and over again. George Michaelson 35:10 Yeah, you know, you sometimes have to accept the favorite model you hold in your heart is just not the one that survives the process. Willem Toorop 35:19 Yeah, unfortunately. George Michaelson 35:20 So that one's out. That scheme has a weakness that is about historical behavior, and you have observed at least some of the other four are capable of healing this behavior. One of them is a delegation in the route to each of the entities, Willem Toorop 35:35 yeah, George Michaelson 35:36 and the other one is the invisible naming. Willem Toorop 35:38 Is the everything authoritative in the root, George Michaelson 35:41 yes. Willem Toorop 35:42 So this is has problems because if the signatures are also added to the additional section, George Michaelson 35:48 yes, Willem Toorop 35:49 then the priming query will become huge for some of those George Michaelson 35:54 Right. So we come up to the primary problem of failing to cope if you are in an older generational behavior, expecting a single packet response, you may enter a problem. However, they are less likely to request DNSSEC. So, for the really old case, fine, but for the intermediate case that still has dependencies but needs DNSSEC, you're now going to. it's a multi-packet response. Does this uplift the TCP? Willem Toorop 36:23 Yeah, we also looked into if resolvers were able to fall back to TCP, and if they would, but it still showed some problems, but there's not a notion, but you can see that if each root server identifier would get its own name, and all the signatures for the addresses would also be included in the additional section that this will become a very large packet, right? Like more than 4k actually. [George: Wow] yeah. So the sixth scheme for which the root server identifiers don't have their own name has much smaller packets, because it only needs two signatures in the additional section, one for all the IPv4 addresses and one for all the IPv6 addresses. George Michaelson 37:09 So we now come to that delicious moment in a podcast that is trying to be about measurement and trying to be about technology in its purest sense. We arrive at layer nine politics. Willem Toorop 37:22 Yes, George Michaelson 37:22 do you think this is a tenable answer for both the root operations community and ICANN in its duty of care? Because this sounds like it has a component of decision that not everyone might be comfortable with. Willem Toorop 37:38 Yes, but there's additional bits of information that I have to give for well-considered choice in this respect, because as long as the names for the root server identifiers or for all the roots is below the root, as long as there's a label which is not the root itself doing this, then there is a possibility that would have been a delegation to those names, right? [George: Yeah], see that. [George: Yes], for that very reason, a resolver cannot assume that the signatures will be there, because it could have been below a delegation George Michaelson 38:21 could have invoked a zone cut and could have invoked the necessity of a glue binding to tell you how to follow the chain. Willem Toorop 38:31 Yes, and therefore any possible intermediate can just take out their signatures without being punished for it, so to say, without George Michaelson 38:40 right, Willem Toorop 38:41 so they don't really add security to the priming query. Yeah George Michaelson 38:49 wow, so we've come to the hard point that security after the fact is an extremely hard thing to do. Willem Toorop 38:57 Yes, and so one of the possibilities is also to change the authoritative behavior, right, because also this open source name server software, which is used by the root servers, can be adopted to just not include those signatures, even though it was authoritatively present within the root, and then those schemes also do not cause any, any problem. George Michaelson 39:26 Willem, I think this is fascinating. It feels like you've conducted a really strong test, a measurement in system with an analogue, a model, and with clean room implementations of proprietary behavior with real-world informed data, and you've been able to do some categorizing of choices faced in the community. I think this is an amazing outcome, but I have a feeling we're going to want to talk to you again about some of the subsequent consequences of this discussion. So, hopefully we can have you back another time to talk about that. Thank you. This has been Fascinating. Willem Toorop 40:00 Absolutely. And thank you for having me. And I would love to come back and indeed talk about the consequences of this research. Thanks. George Michaelson 40:11 If you've got a story or research to share here on Ping, why not get in contact by email to ping@apnic.net or via the APNIC social media channels. Also, remember the measurement@apnic.net mailing lists on orbit is there to discuss and share relevant collaborative opportunities, grants and funding opportunities, jobs, and graduate placings, or to seek feedback from the community on your own measurement projects. Be sure to check out the APNIC website for all your resource and community needs until next time.