Darren Kara - 00:00-00:03 Yeah, you have the long and short of it there, that each one of our pops is insulated the only place that sees your IP address that you know. is the query comes in, we keep it long enough, the IP address long enough to answer your question, send it back to you. You know, so the IP address gets dropped on the floor, but we do count things. We do count, you know, the region where the question came from, we keep the question. We it can associate with an ASN, we can associate it with a locality we use maxmind which I'm sure you're familiar with. Their accuracy is better than what we want so we go in and we fuzz it out When we get a quarry in and we associate it with a region, we don't associate it with, you know, the pin down the street from your house, which is going to look awfully like, you know, George's house. we fuzz it out to the nearest metropolitan area of I think it's fifteen thousand people. So you'll get lost in the aggregation there. George Michaelson - 01:03-01:07 You're listening to Ping, a podcast by APNIC. discussing all things related to measuring the Internet. I'm your host George Michelson. This time I'm talking to Darren Kara from Quad 9. Quad 9 is one of the oldest public DNS services. It was established 10 years ago by Packet Clearinghouse, IBM, Global Cyber Alliance, and Switch. Over time, it has transformed into a public-benefit not-for-profit foundation, headquartered in Switzerland, and serves the global Internet with a twin key focus on the privacy of users And protection against malware and phishing. These dual aims mean that Quad 9 has a data integration burden. To aggregate the threat intelligence feeds and to aggregate user behaviors so they can both protect the communities using them and feedback to each threat intelligence source when their information helps protect a user and which specific threat it ameliorated. But thirdly, the privacy focus means this intelligence has to stay inside the room Nothing uniquely identifying, no personally identifying information, or PII, ever leaves the host and the network at any specific location. It's something they're very proud of And want to maintain, while still collecting and managing, complex information from each site as much as they can. Quad 9 partners with threat intelligence sources worldwide and likes to think there's a virtuous circle of information flowing in and out to mutual benefit, but always under the condition that no PII leaks outward Darren, welcome to Payne. Darren Kara - 02:52-02:53 Thank you, George. I appreciate you inviting me to talk about myself and DNS today, mostly DNS. More interesting than I am, I think. George Michaelson - 03:00-03:04 Well, before we get down to DNS, could you tell everyone a little bit about yourself? Darren Kara - 03:04-03:06 Uh yeah, I'm Darren Kara, I'm principal engineer at Quad 9 which means you know I do whatever I'm asked to do. We're a small shop trying to do good for the Internet. Before that, my history has mostly been in critical Internet infrastructure. Where I met George years ago and looking at the calendar, George, I think it's probably a quarter century since we've become friends and colleagues George Michaelson - 03:27-03:28 Yeah, it was a long time ago. You were working at ARIN at the time, but you've subsequently spent time working with ICANN doing technical deployment stuff there as well, haven't you? Darren Kara - 03:36-03:38 Yeah, you know, early years with ARIN getting used to the the community and running critical infrastructure, you know, in- adder.arpa, whois, managing the registration database. I took a side trip to Time Warner Cable to learn layer two and layer three engineering at scale for data centers and cloud operations. So that opened my eyes as far as, you know, what can be done versus what I thought I knew is about layer two and layer three, you know, when you get with the big boys and get a little bit of education. After my time there, yes, I did go on to ICANN to work in root server operations, managing and operating L root for ICANN, as well as ICANN's internal networks around the globe. So I've had the honor to do some very cool and interesting things in my career. Quad nine was a natural next step from ICANN. You know, I was running a global DNS root server, a global DNS resolver looks very much the same, you know, and the the footprint's very much the same as we had at George Michaelson - 04:41-04:42 So people may not actually realize that Quad Nine has been around for quite a long time. It's over ten years now, isn't it, that Quad 9's been in existence. Darren Kara - 04:52-04:53 Yeah, yeah, right about. Yeah. We're hitting the ten year mark. I wasn't there. I've only been here, you know, eighteen months now, but it feels like home to me. George Michaelson - 05:00-05:01 And this was initially an initiative of packet clearing house, PCH and IBM and some other partners like the Global Cyber Alliance, I believe. Darren Kara - 05:12-05:14 Yes, indeed it was. George Michaelson - 05:14-05:14 But now you guys operate as a independent not for profit and if I get this right, it's actually deliberately structured in Switzerland, is that right? Under kind of Swiss sensibilities about data protection and privacy and that kind of public structure. Is that the model? Darren Kara - 05:34-05:37 Indeed, we we did as you know, as as company matures and you find your own identity, but also the idea of incorporated in Switzerland. to beholden to Swiss laws around privacy was important to us in order to show the community that that's how serious we are, that we would l you know, the rule of law could come down on us if we aren't doing what we're saying we're doing with regards to your privacy. George Michaelson - 05:57-06:02 And that's the signature quality that you guys bring to your public DNS resolver function, isn't it? That you stand both as an anti- abuse bastion and doing filtering and clean feed, but also a really strong message of privacy of end users' use of your technology. Darren Kara - 06:20-06:23 Uh yeah, it's probably what we're most proud of at Quad 9 is the fact that we are here for the end users. We have no shareholders, we have no, you know profit margins and things like that to care about. Our customer is the end user. It's you, it's me, it's mom, it's dad, it's, you know, somebody in, you know, South Africa. It's the world, which is a very nice feeling when I turn up new capacity somewhere knowing that instantly I am helping or can help somebody in a region with their privacy and their Internet connectivity and their DNS. George Michaelson - 06:52-06:53 So how many sites are you guys at now? It would be over two hundred by now, wouldn't it? Darren Kara - 06:58-06:58 So if we had linear growth you know, i i it would be well over 200, but at times you you expand and contract to aging equipment and things like that. Sometimes sites become inefficient because the hardware has become older. We can't get replacements in certain areas and we have to pull out of certain pops. But yeah, we trend up into the right. I think we're right about 200 right now in in, you know, what, a hundred countries or something. So it's the quality, not the dots on the map that really matter. But we do want to be in localities that other resolvers are not in so that we're closer to communities that need us George Michaelson - 07:35-07:36 Yeah. This is going to be quite a challenging conversation given that we're here for measurement, because if you stand by the byline that nobody gets to know about end users traffic through your system That's kind of a difficult space for measurement, isn't it? We've really got to keep ourselves in a narrow band. It's not that you don't measure, because you obviously do, you've got to keep track of your systems. But you're not bringing data out of the room in any way, are you? You don't bring raw packets, connection logs, any kind of individual query burden out of a location. What happens in the room stays in the room. Darren Kara - 08:16-08:20 Yeah, you have the long and short of it there, that each one of our pops is insulated, the only place that sees your IP address that, you know, is the query comes in, we keep it long enough, the IP address long enough to answer your question, send it back to you. You know, so the IP address gets dropped on the floor, but we do count things. We do count you know the region where the question came from, we keep the question. We it can associate with an ASN, we can associate it with a locality. We use maxmind, which I'm sure you're familiar with. George Michaelson - 08:46-08:47 Yep. Darren Kara - 08:47-08:49 Their accuracy is better than what we want. So we go in and we fuzz it out. when we get a quarry in and we associate it with a region, we don't associate it with, you know, the pin down the street from your house, which is going to look awfully like, you know, George's house. We fuzz it out to the nearest metropolitan area of I think it's fifteen thousand people, so you'll get lost in the aggregation there. George Michaelson - 09:10-09:13 But those aggregates are really quite informing, aren't they? You're getting totals that show you the nature of queries. So the thing that we've kind of skipped over is the other half of your story is safety and security of browsing. You're actually blocking and intercepting malware DNS domains and the like as well, aren't you? Darren Kara - 09:30-09:37 Yeah, that's correct in that, you know, we know Quad 9, we kinda live this industry, so we know the trends. I'm always surprised when I run into technical communities that don't quite understand the Quad nine differentiation between us, Google, Cloudflare. Cisco Umbrella, and so forth. And it is, like you said, it's the fact that we partner with 30 plus threat intelligence speeds providers And we take a constant feed from them. So we are as up-to-date as they are with regards to malicious domains and trends. And they go automatically into our block lists. And you get that protection as soon as you point at Quad 9 with your DNS. George Michaelson - 10:09-10:11 If you have a list of block domains, are you also correlating the level of query load that you see coming in per pop that is maybe tickling a particular domain. If I imagine there was a command and control domain for a Mirai attack and it was really heavy in Southeast Asia, would you be able to zoom down and say, oh wow, we have a lot of people who are seem to have been infected and are targeting that class of traffic? Darren Kara - 10:38-10:39 Uh yeah, indeed we do. We and we also have some visualizations around active blocks, which is kind of fun watching in a little explosions on a globe that show activity and it'll show you what block is being hit or we can we can determine what domain is being blocked and what's happening in in that region, in that city, in that country, and so forth. George Michaelson - 10:56-10:58 So what do you actually do? when you get someone coming in and asking for a domain that is healthy and you're going to make that a failing event, how do you signal that back in DNS? Darren Kara - 11:08-11:09 Well, so we do NX domain, we return nothing, we say, We're sorry, you can't get there from here, which, you know, protects you. But if you want For those that are a little more savvy on the call and can are familiar with DNS and can use DIG, when you ask for that query that we're blocking, it's a synthetic response on the resolver, and you will get EDE29. from EDNS extended errors. George Michaelson - 11:32-11:38 The new standard that has more information that can provide additional guidance to why that event fail. So you n-xdomain it. Because I've often wondered if you would do a signal like serve fail, if I had more than one DNS provider and you're my front engine. and you surf fail me, I'm just going to the next guy. But if you NX domain, it's like you've categoricled me it's not there. Darren Kara - 11:58-11:58 Exactly. And that's that is why we do it, so that you don't fall through to your secondary or tertiary resolver. But this improvement with announcing that it's also yes This is a synthetic response to your question. We are being transparent even then that, you know, although we're not modifying the DNS, we're not giving the answer, and that was because of our choice. So it's a way to see it in action. George Michaelson - 12:20-12:21 And you're doing several different address bindings for this service, aren't you? You have Quad 9, I mean it's in the name, but you have variations of the the binding to provide different levels of response, don't you? But Do you engineer these from the same platform? Darren Kara - 12:38-12:41 Yeah, I mean they're any casted from the same nodes, you know, the same pops. You know, you bring up the fact that we do we the the standard Quad 9, you you get what you get, you get our our Threat intelligence, we get the blocking, you know, caching resolver, you get adot, you get adoc, you get you know doh, you get all these great things. Some of the other addresses are 9. 9. 9. 11, which is ECS enabled. I think you probably were gonna be heading there soon enough. And the other one is just straight open resolver nine dot nine dot nine dot twelve, I think. But most people just stick with the the Quad 9 and I've been approached a technical conference is saying You really should have ECS on 9. 9. 9. 9. And we say that we make that available because it's necessary, but we think that has privacy implications, so we don't put it on our primary George Michaelson - 13:26-13:31 Yeah, so for people who aren't up with the buzzwords in DNS, ECS is EDNS client subnet. And that's when I as a client signal to you as the resolver, this is the address I want you to know about me And that can kind of aid with things like geolocation steering. So when people are trying to find closest content from a CDN or specific services targeting their economy or their location. That kind of signal's really important. But you're now hitting that barrier that ECS is phenomenally revealing about me. As the client, right? Darren Kara - 14:06-14:06 Yeah, and and like I said, that's just too close for us. So we are always going to default on the, you know, more privacy is better. We're not gonna leak or bleed any information like that. So that is by choice. So people will choose that, and there are definitely good reasons for using the ECS enabled service, like you said. rather than coming all the way to IAD Ashburn, Virginia to get your content, it will redirect you to something closer. George Michaelson - 14:30-14:30 So you said that you're taking multiple feeds from the threat intelligence community. But are you also supplying data back to them about the nature of DNS queries as they're like a virtuous circle Going on here? Darren Kara - 14:43-14:47 Oh yeah, we we definitely, you know, they get near real-time feedback from us, especially if they start hitting things like false positives so that they can take a look at what's going on in their end because that has infinite amount of value for them, you know, because that's their bread and butter, right? So and we're better to get this information than one of the largest resolvers in the world. George Michaelson - 15:04-15:05 So if you imagine that there's a false positive in their information feed, you would then have to try and compress the cycle time between your data getting to them. them figuring out they need to adjust their feed and you receiving back. So it's kind of a life cycle. It's not like a daily update. Is this a continuous feed mechanism? Darren Kara - 15:25-15:26 It's continuous, yeah. I mean the days of, you know, somebody drops a file, you pull it in and you import it, or those are early days from our career. No, this is all continuous feeds back and forth. As well as we'll get, you know, signaled by the community or even just help desk tickets that say, hey, we got some false positives we think. You know, that being said, if you're using our service and you think you're being blocked. You can check it on our site. You can drop in the URL and say, you know, are you blocking this? Drop it in and hit enter and it'll say whether or not it is in our block list. So that's one way to validate if it's us or somebody else why you're not getting there. George Michaelson - 15:59-16:00 So it's kind of like down detector or is it down for everyone or just me, except it's DNS specific? Darren Kara - 16:05-16:06 Yeah. George Michaelson - 16:06-16:08 You said you're a small crew, you're not a big team but if you've got this many pops and you're running high availability, high reliability service, what kind of systems and tooling are you using to push out code and manage systems? What does this look like Darren Kara - 16:22-16:25 From the the perspective of how we push things out, we're an Ansible shop. So we have tooling around that that pushes out to our pops. Uh our pops are one U pizza boxes, generally speaking, one or two in a pop. Maybe a few more in some of the really high population areas that we hit. It's all free open source software. George Michaelson - 16:39-16:40 For the Anycast thing, you're running a BGP instance on the same configuration you've got virtualized. Darren Kara - 16:46-16:50 Yeah, so we we drop one system in, we throw a hypervisor on it, it peers with our host or provider, you know, whoever's giving us transit. And yeah, we run FRR three range routing daemon in order to do our any cast announcements to the providers. George Michaelson - 17:02-17:05 And I believe you're using DNS Dist as your front-end mechanism in this or is that an older generation? Darren Kara - 17:09-17:10 No, no, we we use DNS just on the front end to handle the connections and, you know, offload them to the back end. Uh resolvers to handle the query. So but you know, also it gives us other things on the front end about DNS tap as well as being able to do filtering and blocking. George Michaelson - 17:25-17:25 Right. Darren Kara - 17:25-17:28 When we do have incidents happening and people Either doing bad things or intentionally or unintentionally. So sometimes the worst abusers aren't malicious. They've just misconfigured something and they just happen to be pointed at you George Michaelson - 17:37-17:38 Yeah. I've used it in limited manners to do parallel multiple DNS queries which saturate the usual providers and we were getting a lot of surface pushback. Because we look like a denial of service point attack from one IP. And it's really interesting how DNS this shards across its back ends. It's really efficient at working out which server is loaded. and how to distribute that balance. It's a very nice piece of code. So you're all for privacy. You've got this feed of bad tr potential bad domains that you're integrating over the data that you collate and aggregate. at each point. What does come out of the room? How are you actually bringing aggregated data out of the room? Are you using a binary feed or are you using JSON tooling? What does this look like? Darren Kara - 18:29-18:30 It's Kafka. We feed into Kafka from each sites after we have dropped any PII and we also do the data enrichment on in the POP. We use product called Vector, I think it's from Datadog. It's very good for our needs and for what we do. So back to the privacy thing. We never do like full log shipping like a lot of people do back to a central location. We don't have terabytes of of data just sitting around. We George Michaelson - 18:55-18:56 No. Darren Kara - 18:56-18:57 Can't afford it. George Michaelson - 18:57-19:01 So you run in some quite bandwidth constrained locations, don't you? So your management channel, you would have to be fairly conservative about how much data you're pumping down this. Darren Kara - 19:07-19:11 Like I said, as time has rolled by though, our sites we do have kind of minimum viability that has to happen. If we can't get the telemetry out of it, it's not really a viable site and we need to investigate a new host, new provider, things like that, in order for us to get the best picture from you know our constellation of pops around the world. As much as we want to be everywhere, we do have to have a a minimum set of requirements in order to be there. George Michaelson - 19:33-19:35 But you're kind of now thinking about data and measurement. You're now in this world where if you can't bring the full packet stream out of the room. You can't really do what if out of the room. If you decide there's something you're interested in, you've got to go back into your Ansible configuration, tell the node that you want to start aggregating against a new data object. and get it exported back up your Kafka feed. So it's a little bit heavyweight to start measuring new things, isn't it? You've got to think about it. Darren Kara - 20:03-20:04 I think the heavier weight there isn't JT, our our CTO, is really good with this. It's more of trying to figure out exactly what we have. We have a lot of data. And it's about where to go and what to look and you know, what to aggregate and what to correlate, I think is the bigger question there. George Michaelson - 20:18-20:18 Yeah. Darren Kara - 20:18-20:19 Even after ten years. There's just some crazy data in the resolver space and every day you come up with a new way to to look at it or make associations with the data gives useful information that might be able to be given back to people to use in a meaningful way to help protect people. George Michaelson - 20:35-20:36 I'm continually amazed at the things that you can find in DNS, the number of malformed queries that I used to see looking in the APNIC feeds. People would be doing reverse address lookup, but they'd have misconfigured things and they'd be appending an entire forward domain lumped on the back of the inandra. arpa. I see people doing lookups for ULA addresses that were never meant to leave the host, but they were basically telling me The prefix they were using inside their local site deployment, you see some truly amazing things in the DNS Domain stuffing with more than sixty three labels chained up together. You see crazy things coming out of there, don't you Darren Kara - 21:20-21:26 Well you remember with root servers before a lot of the cleanup happened and things like that that there was just trash. I wanna say that something kept asking the root servers for dot balcon, I think was one of the yeah strangest things that percolated up into the top ten going. What okay. George Michaelson - 21:36-21:36 Who knows? Okay, you want to know about Belkin? That's fine. Ask away. Ask as many times as you like. Darren Kara - 21:42-21:42 No, it it's there was some mistake on some hardware that was out in the wild. George Michaelson - 21:48-21:48 So with the ECS, you're saying that Maxmind is too accurate. D is there an automatic way that you can dial it back to a lower granularity setting? Or did you have to construct this for yourself? Darren Kara - 22:01-22:03 Yeah, no, we we take it from Max Mine and then we start cutting out. Our our CTO handles that too. He's a very much a a data minded fellow and is very good with that, but he goes in and makes sure that we have less accuracy. You know, and so he cuts it down from five hundred megs to one hundred and fifty megs. I can't remember the current size of the MaxBind database. George Michaelson - 22:20-22:22 Other people, other providers cooperate in this activity called Day in the Life. which is like a single point capture of DNS visibility. That's not something you guys can really entertain, is it? That's just not going to be part of your activity Darren Kara - 22:37-22:37 I did that at ICANN as a root server and that's definitely something in their purview. But no, we we don't save the logs in the way that they need us to to ship them over and yeah properly do their thing. We are heavily invested in going to DNS OARC and participating there with all of our colleagues. We tend to go to there and ITF and other DNS heavy conferences, RIPE, we attend George Michaelson - 23:01-23:02 Yeah. You've recently taken a role in NanoG programming as well, haven't you? Darren Kara - 23:07-23:08 Yeah, I'm doing a little bit of work at NanoG as far as community engagement. George Michaelson - 23:12-23:16 But you're doing some outreach and engagement in that space. Darren Kara - 23:15-23:19 Yeah, that's more of a personal thing for me, giving back to the community. Like we talked about, it was one of the places when I first started Aaron that I got to go and meet the larger community and people like you and other operators and things like that. And it just opened up my career. I'm allowed to participate more now that I work at Quad 9. George Michaelson - 23:33-23:37 It's nice to work for an employer that's open-minded about a community dialogue. So I have a feeling from looking online that your map of distribution of nodes is quite heavy in Europe and America But you're not actually so visible in the Asian footprint, are you? You don't have that many nodes over our side. Darren Kara - 23:53-23:54 I I would have to take a look, you know, we're always looking for more places and we have a good partner in Australia. Swoop yeah I think we're at five or six sites with them right now and so they've been great. George Michaelson - 24:03-24:05 But you'd like to boost your footprint? You'd like to see more coverage in our region Darren Kara - 24:07-24:13 One of the things we need from your region, it's it's less about footprinting, but yes, if people are out there and they want to partner with us and give us transit, rack space, server, and and all of that, we're always happy to engage with with people. More importantly though, we need threat intelligence providers, they're specifically APAC or Australia. George Michaelson - 24:27-24:27 Uh Darren Kara - 24:27-24:31 we only have a couple of our thirty five that are from that area. We don't always look for the biggest threat intelligence feeds, sometimes there's curated ones that are regional or country based or industry based for a region. And if people are listening and they're interested and wonder, you know, hey, we might be able to help Quad 9 in that fashion and we have this feed that we use locally in Queensland or something like that. And we would be interested in partnering with Quad 9. They can come to our website and I think it's probably partnering at Quad 9 that you can send an email to and we can have a discussion. Our threat intelligence people here will reach out and see if some arrangement can happen. But we're always looking for good TI feeds and they don't have to be huge. They just have to be well curated. We just looking for people that to know quickly what the malicious domains are and things that are spinning up anywhere in the world in a region and so forth like that. George Michaelson - 25:22-25:27 So that would come from things like local resolvers that people are running in region? What kind of facilities typically generate a log stream that you can then use to do your work with? Darren Kara - 25:33-25:35 We don't tell people how to collect data and send data to us. If they're already a data provider, a threat intelligence provider, those are the people that we're looking for. George Michaelson - 25:40-25:48 Maybe there's an opportunity for you guys to come and talk at an apricot meeting or an APNIC meeting and see if you can build some engagement. Darren Kara - 25:48-25:49 We would love something like that. Our constraints though obviously are financial. You know, we are a not-for-profit, you know, as as Blanche said, we get by on the kindness of strangers. It's interesting, very thin model. We do get grant money though to do things like outreach and to visit places. Grant money's kind of crazy and interesting. You know, a lot of times it tied to a specific thing. You need to go to this conference in that country. and talk about privacy or privacy as you would say. George Michaelson - 26:16-26:20 Are you making a comment about my English accent? Darren Kara - 26:20-26:21 I like your English accent. I I wish I could say privacy with a straight face George Michaelson - 26:24-26:24 It's okay. I struggle saying router. It's a router. Darren Kara - 26:27-26:28 It's a router. Yeah. George Michaelson - 26:29-26:32 Is there a service people can come to on Quad nine's pages that gives them some sense of what's going on. Is there like a visual display people can come and look at? Darren Kara - 26:39-26:41 Like I said, we have this globe that spins that is not quite interactive yet. It takes resources. So But you see little explosions about where things are blowing up and and internally we have much better view of being able to watch this, you know, you put on some ambient music and put it on another monitor. George Michaelson - 26:55-26:56 And if different threat intelligence companies are somewhat competitive in that marketplace of ideas You guys are kind of sitting neutrally in this, aren't you? You're not advocating for any individual source or preferencing. You're a clean player with all the different sources. Darren Kara - 27:15-27:20 They all get a spot on the sponsor page and you know we list all them, but there's no priority or no, you know, these guys are Better than those guys. So yeah, no, we're not in that business. George Michaelson - 27:25-27:29 And if I imagine I provided you with a feed of things I was seeing Could I be getting back from you some idea of hit rates that you see in Quad 9 against those labels? Can I sort of proof check my own feed in your data? Darren Kara - 27:40-27:44 It's that symbiotic relationship, it's that life cycle, they are getting it back. They when we block something and we've received it from their feed, they do. They get counters back. showing how many times it's been blocked and so forth. So very tight relationship. They only get information about blocks from the list that they've provided us, not from any other threat intelligence providers. George Michaelson - 28:02-28:03 Right. Everyone's clean. You don't have leakage between the sources. You would be seeing a whole range of different modalities of access. It's not just UDP on port 53, right? You see TCP traffic. and the new DNS over HTTPS traffic. Are you not concerned about your scaling as traffic packets increase in size? DNSEC and other payloads increase the size of the query load? Are you guys having to rescale? Darren Kara - 28:33-28:34 Not yet. Our POPs seem to be handling everything okay. We continue to embrace new technologies and new standards out of the IETF and try to be a leader with regards to the privacy related ones like authenticated DNS over whatever So we aren't seeing a crushing amount of change. You know, that being said, you know, handling all those things when there is, you know, like I said, it's it's not always malicious. Sometimes, yeah, sometimes you get attacked, but Sometimes it's the misconfiguration. I turned up a pop the other day and we keep track of surf fail and you have this nice little graph. And, you know, there's naturally an amount of surf fail from any pop. I mean, it's it's just natural. You just don't want it to be past a few percentage points. If you can't understand why. And this one was like 20% serve fail. And I'm freaking out because I was like, what's going on with the routing to this pop? What's happening here? Why am I served failing? George Michaelson - 29:30-29:31 Yeah. Darren Kara - 29:31-29:32 It ended up all being because of one authoritative server half the world away that wasn't answering all the queries for domains that were being asked of it, right? That it was authoritative for. And it just kept spiking our serve fail graph. And when we finally realized it was just one. George Michaelson - 29:48-29:49 It's someone else. And so you were actually measuring failure in someone else's authoritative framework. Darren Kara - 29:55-29:55 Yes. And we reached out to them and we said, you may want to fix this. When we see really huge anomalous behavior like that and we can identify Who it is, we do the right thing and we reach out and say, you might want to check why your DNS rotative server is falling over and why you only have one. But There is a lot of data that can be plunged like that also from the Quad 9 data and latency between authoritative servers, CCTLDs, GTLDs, and things like that. root servers. One of the reasons why we aren't doing hyperlocal root is because we would rather get the measurement between us and the root servers. George Michaelson - 30:30-30:32 I'm quite a fan of local route. I think it's a good idea and I'm operating one where I am. But I can understand that you specifically might step back from that, because you really want to see the traffic both sides of the deal, don't you? Darren Kara - 30:44-30:52 Yeah, no, that is more important to us, I think, because you know, I have a a ticket somewhere in my ticket queue that says investigate and perhaps do hyperlocal route. It's kind of the opposite of what we want. We want measurement. George Michaelson - 30:54-31:00 Yeah, you want to know the variance in the back and you want to understand aspects of that because not everyone is going to have a local route. So you kind of want to be exposed to the behavior to understand it Darren, that's been absolutely fascinating. Thank you for coming on ping and talking about the setup. Darren Kara - 31:12-31:13 I appreciate you having me, George. It's always great talking to you and I'm happy to be with a company where I'm well loved when I go to conferences and I don't mind proselytizing about Quad 9. It's good to be. to get on your show and especially the APAC region being represented. We'd really like to hear from more people in that region and partner with them. So please reach out. To Quad 9 if you think you can help in some way with the very, very important work that we do. George Michaelson - 31:38-31:38 Super. If you've got a story or research to share here on Ping, why not get in contact by email to Ping at APNIC or via the APNIC social media channels. Also, remember the measurement@apnic.net mailing list on orbit is there to discuss and share relevant collaborative opportunities, grants and funding opportunities. Jobs and graduate placings, or to seek feedback from the community on your own measurement projects. Be sure to check out the APNIC website for all your resource and community needs. Until next time.