Duane Wessels 0:00 There is this jump that you're talking about, the 5011 jump, 30 days after it goes from I don't know, very small percentage up to like maybe 80% or 90 85% in just a day or two. But that remaining 10, 15% takes a while for them to get it, and those could be resolvers or other DNS software that does not implement RFC 5011 for whatever reason, right? It could be simpler software, and so I think we realize that you know not everything out there will implement RFC 5011. George Michaelson 0:37 You're listening to Ping, a podcast by APNIC discussing all things related to measuring the Internet. I'm your host George Michaelson. This time I'm talking to Duane Wessels again. Duane is from Verisign, and we're talking about completion of the most recent DNSSEC KSK roll, which started in 2024. Duane and his co- author Roy Arends from ICANN have published a two-part blog series at Verisign and ICANN discussing the experiences of the two DNSSEC key signing key or KSK roll-over. I say the most recent KSK roll started in 2024, but there's actually a twist to the story that we discuss in this episode. Sometimes plans don't work out how you expect, and this time there were two reasons for the role to happen in 24: the global pandemic and a completely separate hardware supply chain issue. It's worth understanding a bit of history here. DNS wasn't designed with security in mind. The cryptographic system deployed in the DNS is very similar to website public-private certificates, except there's no certificate. In the DNS. There are just the two keys, public and private. A special key pair called the key signing key or KSK is used to sign the zone signing key or ZSK. The KSK is both used and changed less frequently than the ZSK. Signing the KSK at the very top of the DNS is quite a big deal. It establishes what's called the trust anchor in the public-private keying system. You want to make sure it's distributed widely and signal change as clearly as possible. The DNS KSK was first deployed in July 2010. There was an intent to change it in a reasonable number of years, but for a number of reasons, this didn't happen until the first change of the top level key happened in July 2017, seven years was definitely too long. To be fair, we'd never done something as significant and visible as this before. Key roll-over aren't fast. This transaction completed in October 2018. With experience under our belts, the DNS community aimed to commence the next key roll-over in 2023, improving on the seven-year wait from the first time, and this is what Duane and Roy have been blogging about. Duane, welcome back to Ping. It's great to see you again. Duane Wessels 3:16 Thank you very much, George. It's very great to be here. George Michaelson 3:18 You've recently been writing again on the Verisign blog with your co-author Roy Arends from ICANN. I actually picked it up from Circle ID, who I think did a republication. You're writing kind of like a second part story on the key roll in the DNSSEC route. That's right? Duane Wessels 3:39 That's right. Yeah. So Roy and I wrote together a blog post published both by ICANN and Verisign to talk about what to expect a little bit for the upcoming KSK roll-over, which happens in in October. George Michaelson 3:52 We should probably roll the clock backwards a bit and kind of cover a bit of history here. DNS, one of the oldest protocols in the global Internet had no baked-in security framework, and there was a long design exercise to come up with a way of doing public private keying in order to get signatures over the state of the DNS. That kind of took two forms. There's the long-lived key, the KSK, the key signing key, and that signs over much more frequently used, possibly shorter-lived keys, the zone signing keys, and the part of the game that we're talking about here. It's the KSK part, isn't it? Duane Wessels 4:33 Yeah, it's the KSK part, and for the root zone, it's particularly interesting, in my opinion, because unlike many organizations that might deploy DNSSEC; those other organizations may operate or may do both KSK and ZSK operations. But it's the root zone. ICANN has the responsibility for the KSK, and Verisign as a root zone maintainer has responsibility for the ZSK, the zone signing. George Michaelson 4:58 We have the deployment starting. In mid year 2010, we went too long, didn't we? I mean, we didn't get to a point of achieving our first key roll until about seven or eight years later. Duane Wessels 5:09 Yeah, it was definitely interesting when the root zone was initially signed, like you said, starting in 2010. I think in most people's minds, most people who were involved, the idea was like after five years we'll do a key roll-over, but something happened in 2014, which kind of made it necessary to postpone that. Which was the IANA stewardship transition, right? Duane Wessels 5:29 So that's one reason why that first KSK roll-over didn't really start until well, it was scheduled for 2017. George Michaelson 5:35 If we also kind of come to this point, we hadn't really thought through technology to make a key roll effective and easy to do. When you're operating in most PKI public key cryptography windows, you kind of have an expectation. You're going to send people out into the field with new trust anchors. You're going to do a bunch of stuff. The DNS, we were looking for automatic mechanisms, and we had to come up with a standard mechanism for signaling, we're about to enter this window. So that's 5011? RFC 5011. Duane Wessels 6:07 Yep, yep. George Michaelson 6:08 And it imposes quite well defined time windows when you should expect to see things happen. So in the case of that first key roll, you actually were doing quite a lot of measurement, weren't you? You were standing to look at the behavior of the systems and get a handle on what was going on? Duane Wessels 6:27 Yeah. So as you said, RFC 5011 is the one that describes how resolvers should adopt a new trust anchor that gets published in the DNS. I don't remember the year, unfortunately, that RSC 5011 was published off the top of my head, but something that was sort of an afterthought, I guess, in my opinion, for operating you know things like the root zone was there was no way for anyone to know which trust anchor resolvers had picked up, and so myself and others - George Michaelson 6:56 yeah, it was the outbound signal from the key authority to say I'm going to make a change, and the missing bit here is a resolver being able to say, "Hello, This is the one I'm looking at." We didn't have that mechanism, did we? Duane Wessels 7:09 Yeah. So another RFC 8145 was published in 2016 2017. George Michaelson 7:16 Yeah, 2017. Duane Wessels 7:18 So that was kind of a late comer to this whole process, but we did get some data from that got deployed in resolver software, and we started getting data from resolvers about which trust anchors they had configured. George Michaelson 7:31 Yeah, your first blog that you wrote after the first key roll, reflecting on things, there was a bit of an anomalous signal popped up in the way people were using the technology. Duane Wessels 7:43 yeah. So this trust anchor signaling technique is -it's not perfect. It has some complications, some things that you have to be aware of when you're looking at the data and thinking about how it works. One aspect of it is that it's not secure data in itself, right? So there's nothing in the signal that can tell you that it was an authentic signal, meaning somebody could inject false data if they wanted to. That's one problem. Another point that people raised was, you know, it's very common these days that homes and offices have many devices operating behind single IP addresses, and so you get a signal or maybe two signals or more from a single IP address, and they could be saying different things. They could be mixed signals, right? George Michaelson 8:18 Yeah. Contradictory. Duane Wessels 8:20 They can be contradictory, yeah. And so, in this protocol, there's no real way to sort those out, and so that's something you have to just be aware of. George Michaelson 8:26 We had that first key deployment 2010. We have the roll delayed because of some procedural aspects of structural governance in ICANN. We get the roll started in 2017, and we have this defined timer. And in the modern Internet world, people are routinely talking about transactional change occurring in microseconds. You switch between alternate DNSs in microseconds. You fall back to another database server in microseconds. We do not talk in microseconds in things like key roll, do we? Duane Wessels 9:00 No, we don't, and I think the reason for that really is that it's very common that DNS software, your BIND, your Unbound, those kind of things, they get updated on pretty long life cycles, right? Yeah, it can take quite a while for software updates to percolate out through all the places that they need to go, and that is one reason why things in the DNS don't happen, as you said, on microsecond timescales, George Michaelson 9:23 They don't even happen on day timescales because the initial signaling window of interest is a 30 day timer to see clear signal of a change in behavior. But the actual duration of the key roll spanned out over more than a year, didn't it? That first roll, it was 15, 16, months? Duane Wessels 9:43 Well, we have to talk a little bit, I guess, about the different phases of the key roll-over, right? So they're lettered phases A through G. The interesting ones were phase D George Michaelson 9:54 seven. I had to do the count on my fingers to remember the span here. Duane Wessels 9:59 So phase D. Was the phase when the new key first appeared in the DNS, right? Yeah. And then phase E was the phase where the signing switched from signing with the old key to signing with the new key. And in the schedule for 2017, there was one calendar quarter between those phases. So the expectation was the key gets published in the root zone. 30 days later, the resolvers see it and do their RFC 5011 dance, and they pick it up and they adopt it. And then another 60 days later, you actually do the key roll. And I think in hindsight, that was a little bit of an aggressive schedule. The current key roll-over is much more slow than that. George Michaelson 10:37 So we had this first roll. We had phase stages. It takes months to complete, and we have a window in the in between period where we're actually sending larger data because we have to have a crossover period where we're sending in both old and new. And in some ways, the critical time, the one that we're actually approaching now in this second roll, is when that first signature goes away, and what we're left with is the new signing. Right? It's actually the drop off the other side. Duane Wessels 11:06 Yeah. The tricky part about that, or any key roll-over, is when you flip them and you start signing with the new key. What is the behavior of the population of resolvers out there? You know, did they get the new key? Are they going to have resolution failures? George Michaelson 11:17 I really strongly recommend people listening to this podcast, go and have a look at either the old and the new blog, the first and second blog, or the second blog has these beautiful timing diagrams that you collected. Now, in strict sense, you didn't collect this data in your functional role as the operator of this technology, did you? This was collected through a different functional role that Verisign has? Duane Wessels 11:43 Yeah. So the Trust Anchor signaling data, the way it works is a resolver sends a signal in the form of a DNS query to a name server authoritative for the zone that Trust Anchor is for. So in this case, these queries go to the root zone, and Verisign as root server operator receives those queries, and that's where we were able to collect it and analyze it and make these graphs. George Michaelson 12:05 But that's a kind of separate functional role to being the party that's contracted from ICANN to perform operations using the key. We really do have to understand there's a separation. Duane Wessels 12:15 Yeah, technically those are those are different roles. Verisign is both the Root Zone Maintainer, which is our role as the ZSK operator, but we're also a root server operator running the actual services for the root zone. George Michaelson 12:34 These charts, and truly, folks, they're worth looking at. It's the most amazing uptick signal for that 30 day window when 5011 first kicks in. But you then had a quite long persisting tail. There seemed to be a population of people that just didn't quite get the signal. Duane Wessels 12:42 Yeah, so there are. That's right. There is this jump that you're talking about, the 5011 jump, 30 days after it goes from I don't know, very small percentage up to like maybe 80% or 90, 85% in just a day or two. But that remaining 10, 15% takes a while for them to get it, and those could be resolvers or other DNS software that does not implement RFC 5011 for whatever reason, right? Could be simpler software, And so I think we realize that you know not everything out there will implement RFC 5011, and there are other ways that trust anchors could be up. George Michaelson 13:12 Yeah, and it's kind of interesting when you think about the community burden that Verisign's taken on providing this facility. It's very easy for people in my role with no strong responsibility to kind of look at 80/20 80% 85% and think "yay job done go home" and you guys are sitting there thinking wow we have 2.5 to 3 billion users and if 15% of the global community is sitting behind a service that hasn't seen this. That's a really astronomically large number of people. You can't be so casual, can you? Duane Wessels 13:46 We do take that pretty seriously. You know, one of the real challenges we had with that roll-over in 2017 was that this trust actor signaling technology was really pretty new, and so the sample sizes were relatively small. You know, we're talking less than 1000 source addresses per day. We're sending these signals, and so there was a lot of questions about, you know, is this representative? How many end users might be impacted by this? As you said, George Michaelson 14:12 Yeah. Let's come to the second key roll. Seven years was too long, so there's kind of a vibe out there in the community. Let's do this more often. Roll frequently, and the model settled on kind of three years. Is that right? Duane Wessels 14:25 Yeah, yeah. IANA did a community consultation, developed this plan. It was a three-year cadence to the key roll-over every three years. George Michaelson 14:32 Yeah. So we get three years up from 2017, 2018. Right. We hit a world event. Right. I mean, this is not the year that you want to start undertaking some technology change. COVID happened, and part of the behavior of this keying stuff is it's like a public engagement. It's a public performance, a ceremony. It's called the key ceremony, and it's inclusive of people from all around the world. Coming in and visibly participating in that keying process, you have a role in this, Duane, don't you? Duane Wessels 15:06 Yeah, Verisign has a role. We go there and verify what we call the key signing request. Yeah, normally everyone was traveling to these meetings four times a year. and the pandemic really changed that. George Michaelson 15:19 So schedule gets put back, and then we hit a second event, which is a little less predictable and understood. Keys they don't just sit on floppy disks attached to a laptop, right? Keys actually have to live in secure structural devices called hardware security modules (HSMs) and the way the key signing for the DNS root works, you have two facilities, one east coast, one west coast, and they run identical hardware. And you'd selected a device. The community had kind of come to that device as the target to run for this function. Duane Wessels 15:56 Yeah, in fact, so ICANN as the KSK operator had selected HSMs, as you said. George Michaelson 16:02 Yeah, Duane Wessels 16:02 Verisign also used HSMs for the ZSK in our facility. So yeah, both parties were using those those HSMs. George Michaelson 16:11 HSMs they have this set of levels you can operate in, and you really do want to know the implications of the level you run in. And I believe the KSK operates in FIPS, so Federal Information Processing Standard, FIPS 120 Level 2, and it's a mode which allows you to take a key and send it to another of exactly the same generation using special keys to protect it in transit between the two machines, and that's how you can have East Coast and West Coast both operating. But you really cannot get those keys out and into a new machine, can you? That's not on the table. Duane Wessels 16:50 Yeah, same vendor, like you said, you can do key export and import, but between vendors, it's not supported, and it may be out of the standard of. George Michaelson 16:57 A second thing that I've personally experienced operating HSMs is that the company that operates the HSM has to be in a long-term relationship with you because you have to have some certainty about spare parts and also about hardware and software generations. And we experienced our vendor coming to us and saying we can't commit that the unit you currently have will be supported if you buy our current technology. You've hung on to it too long, so the idea is you really have to keep in lockstep through the generations that you're running with. And I believe the signal that came was the hardware technology you guys have got to operate this system is not going to be available. Duane Wessels 17:39 Yeah, it was a little bit surprising because ICANN and Verisign had just restarted the KSK roll-over process. The key was generated in 2023. We called it KSK 2023. It was generated on those HSMs, and then you know just a month or so after that key got generated, there was an announcement from that vendor that they were going to make those end of life. They were not going to support them. I think past 2025 was was the date they gave. George Michaelson 18:04 So 2025 is inside the foreseeable lifetime of the keys that you were generating. I could handle someone saying a year after end of life we won't be there. You could kind of go "Fine. We'll keep on this course", but with the hardware being end of life inside the life of the key, you really had to reconsider, didn't you? Duane Wessels 18:22 Yeah. So at that point, there were no great options, but basically, that KSK 2023 had to be abandoned, and both ICANN and Verisign had to find new HSM vendors to bring in test and evaluate and all that stuff. George Michaelson 18:38 And these are pretty heavy weight tests too. My experience of going to the keying ceremony is that there's absolutely no transactional moment in that ceremony where you guys haven't sat outside and tested it in the hardware you have available before it actually goes live on the day. Everything has to be seen to work. Yeah, we come to the second key roll. It's 2023, and we hit this problem where the unit that you're using reaches its end of life, and because of that, you have to start looking at the duration of the keys you're likely to be creating. And you had this crop up. Duane Wessels 19:14 Yeah, you're right. So in 2023, as you said, we had just sort of resumed the process. Keys were generated on those existing HSMs, and it wasn't more than a month or two after that where the vendor for those HSMs announced that not only would the model be end of life, but they were not going to support that model in the future. I think they give a deadline of 2025 or so George Michaelson 19:36 ... Inside the lifetime of the keys that were being created. So - Duane Wessels 19:40 Yes, definitely, George Michaelson 19:41 Yeah, and you basically had to reset and start again, didn't you? With ICANN, you guys had to go back into tender process, identify new vendor source equipment, and then start the whole thing from scratch because you can't take the key out of one manufacturer and bring it into another. Duane Wessels 19:58 That's right. So both ICANN and Verisign. We're using the same HSM vendor for the root zone. We had to identify new vendors, new HSMs, and just said do all the feature testing, evaluations, and whatnot. And so that basically delayed the roll-over for another year. George Michaelson 20:14 Yeah, we actually are already in the time cycle, the window when we're performing this key role. It started in 2024, didn't it? With publication of the key visibly into the zone. Duane Wessels 20:26 Yep, the keys were generated in 2024. You're testing my memory about publication of the zone. It may have been in 2025, George Michaelson 20:34 And we've arrived at the back end of those seven lettered sequences, October of this year, we're going to see dropping of the old key. The 2017 key is going to be removed from the signing set. Duane Wessels 20:50 Yes, on October 11th, the root zone will have a DNS key RRSET that is signed by the new KSK, and the old KSK will still be post published for a while, but it won't be used for signing. George Michaelson 21:03 This thing that I really loved in your second blog, the piece that you and Roy from ICANN Roy Arends wrote, is that you've done exactly the same timing diagram, haven't you? You ran the same charting method. Duane Wessels 21:16 Yeah. So we've been collecting this data all this time, and every now and then taking a look at what the trust anchor signal data says for the blog post. We generated a graph that lines up the publication dates of the two keys: the 2017 key and the 2024. George Michaelson 21:30 It's like a giant time signal uptick that you can align the two data sets on. Duane Wessels 21:36 Almost like a step function. George Michaelson 21:37 Yeah. So when you did the first round, you had that 15, 10, 15 percent gap at the top mark that hadn't quite got with the new program this time round. The signal is quite a bit better, isn't it? Duane Wessels 21:49 Well, for one thing, we have a lot more sources of the signal now. We have something like 160,000, which is really good. But what's interesting to me in that graph is that, well, in 2017, there was this sort of anomalous - George Michaelson 22:01 Oh yeah, there was a weird dip in the middle of it. Duane Wessels 22:04 Dip in the data, which I have, to be honest, I've never been able to fully explain. But if you sort of put that aside, otherwise the two graphs line up very nicely. There's this huge step function jump, and then after that, there's kind of this slower increase, ticking up and up a little bit day by day by day, eventually getting up into the range where you know now we're at like 95% or so of sources have the new key George Michaelson 22:25 95 that's a lot better signal than last time round yeah yeah and in October the final transitional step happens and you come back to a single signing outcome using the new key and we're looking at about a three-year window before we enter this cycle again. But next time round is going to be a bit different, Duane Wessels 22:49 But to be clear, not three years until we enter the cycle, because actually we enter the cycle right away. It's three years until the next roll. Both Verisign & ICANN teams have a lot of work to do almost immediately after this roll-over, to get ready for the next one, which, as you said, is going to be an algorithm transition. George Michaelson 23:06 Key roll is when you say, using this cryptographic system we've got, I'm going to use a new key. Algorithm roll is when you say I'm going to use a new system to design keys and to do the encoding and decoding. What's actually going to be happening in this algorithm change? We're going from RSA to ECDSA. Is that it? Duane Wessels 23:30 That's the plan to go from RSA to ECDSA. Currently, for the root zone, both the KSK and the ZSK are RSA 2048 bit keys, and then once we're past the algorithm transition, so they will both be ECDSA keys. But due to the way algorithm roll-over work, in the middle you have this period where you're double signing. Basically, you have to sign with both algorithms. George Michaelson 23:54 Yeah, so we're going to have slightly different behavior in this next cycle of key change because we're going to have larger packets happening, and we're also going to be seeing slightly different behavior with this new key algorithm. I noticed in the recent blog that both you and Roy have been very careful to give people a clear signal if you want to know from looking in your local configuration that you have the right key. Here is how you can see the key. The key actually carries like an identity number, doesn't it? When it's sent to you. Duane Wessels 24:29 In DNSSEC, keys have tags. We say key tags. It's a 16 bit identifier, a number between zero and 65,000, usually in decimal, sometimes hexadecimal. But you can find that key tag in the files that the resolvers read and write on their system. George Michaelson 24:45 So, if you're worried or are interested, and you'd like to understand your own resolver, the blog actually helps you where to find the file and where to find the key tag and what key tags you should be looking for. And when we come to the algorithm. Roll, you'll be able to do the same thing as well. Duane Wessels 25:02 Yes, we should be able to do the same thing. Yeah, it should be a very interesting project. George Michaelson 25:06 It's going to involve slightly larger key ceremonies as well. I think I'm probably going to be attending two ceremonies next year instead of one because ICANN have said they'd like to get all of the holders of fragments of the securing technology in the room to perform the transition into the new algorithm. In some ways, this new key signing has been a slightly more relaxed activity. Duane Wessels 25:31 I attend the key signing ceremonies as well, as you said, and I find them really fascinating. There's a lot that goes on more than just even signing keys. I mean, there's all this maintenance of HSMs, acceptance testing? as you said, there's these recovery key shareholders. Those shares should be exercised every now and then to make sure that they still work. and so what originally seemed like when the root was first signed, the ceremonies were I don't know a few hours long. They've sort of grown to longer things that even take sometimes more than one day. You know, they'll split the activities over a day or two. George Michaelson 26:05 Yeah, an an interesting quirk is that when you've finished with an HSM and with a key, you're kind of sitting there holding a a box, a small pizza box, and you look at it. What are you going to do with this machine? It's really not a good idea to just let it go out into the world. So one of the things we're going to be doing in this next coming ceremony is taking those older HSMs and very carefully breaking them. Duane Wessels 26:32 Yeah, taking them apart. They're designed so that when you take them apart, they should stop working. And so that's something that we can actually test. George Michaelson 26:39 Yeah. So we're going to come out the other side of this with some physical components that we have to deal with, but we've maintained a kind of zone of security around all of the trust of these private keys. Duane Wessels 26:51 Well, I'm looking forward to seeing you at one of the upcoming ceremonies. George Michaelson 26:54 Yeah, thank you, Duane. This has been really fascinating. We'll put links to the blogs on the Verisign website, in the article that goes up with this podcast. Duane Wessels 27:04 All right, thank you. George Michaelson 27:05 Thank you. Duane Wessels 27:06 All right, see you next time, George. George Michaelson 27:08 If you've got a story or research to share here on Ping, why not get in contact by email to ping@apnic.net or via the APNIC social media channels? Also, remember the measurement@apnic.net mail. list on orbit is there to discuss and share relevant collaborative opportunities, grants and funding opportunities, jobs and graduate placings, or to seek feedback from the community on your own measurement projects. Be sure to check out the APNIC website for all your resource and community needs. Until next time.