Doug Madory 0:00 So we, I guess we first did this in 2022, and it was, again, it was very clear, I had already kind of started encountering this anyway, was if a route is invalid, it just doesn't propagate very far. How far it propagates really depends on who are the upstreams. And it becomes a complex it's very hard to predict, but at very least it's going to be, you know, you, you'd be really hard pressed for starting anywhere on the Internet to get a half a propagation of half the Internet within a valid route. Just can't go very far. George Michaelson 0:36 You're listening to ping, a podcast by APNIC discussing all things related to measuring the Internet. I'm your host. George Michaelson, this time I'm talking to Doug Madory from Kentik again. Doug has been on ping before discussing his work measuring RPKI, as well as a presentation he gave to the pulse Internet measurement forum, or PIMF. Doug and I discussed how he measures RPKI uptake worldwide using a combination of BGP analysis, data available to Kentik from its own tools and data collections, the Oregon route views collection and the RPKI published states, how has the world moved on in RPKI uptake? How are both the produce side and the consumer side, outcomes looking across 2024 Doug reviews the state of play and his sense of secure BGP in the current Internet. Doug, welcome back to ping. I think it's been about a year since we were last speaking. Doug Madory 1:37 It has George Michaelson 1:38 So last time we spoke you were doing work in RPKI, the resource. PKI, we were at the apricot meeting in Bangkok, [Doug: yes] and you were presenting on some interesting things you were seeing in global BGP, at that time, you've carried on this body of work, haven't you? Doug Madory 1:57 Yeah, this has been an area of interest of mine for the last three years of trying to come up with some interesting perspectives on the rate of adoption of RPKI. ROV, George Michaelson 2:06 so just to recap a little bit. RPKI, the resource. PKI, this is cryptography applied to Internet number resources, and the primary purpose here is to present proofs of behavior in BGP. And it's a classic producer consumer problem space, right? Making things me as an asset holder, speaking BGP, I make ROAs route origin attestations. That's my assertion of what I want the world to see. And ROV route origin validation. That's the consumer side of the problem. That's people actually choosing to look at BGP, look at the things I've said, and make decisions whether they believe what they're seeing. Are you looking at both of these things? Doug Madory 2:49 Yes, I am. George Michaelson 2:50 Can we talk a little bit about the shape of the world in this Can we start on the production side? Doug Madory 2:54 Sounds good. Yeah. George Michaelson 2:56 So how do you construct this model? What are you using as your information sources. Doug Madory 3:01 Obviously, we're using BGP data. I make a lot of use out of the route views, public BGP repository. George Michaelson 3:07 So we talked a little bit about that last time. Route views is a project running from the University of Oregon, [Doug: yes], and they have multiple points of view in the global Internet, collecting [Doug: hundreds], hundreds, and they're collecting the state of BGP, but Kentik also are building your own internal models of how things are done, aren't you? Doug Madory 3:25 Yeah, so we I use the route views data. I have some my own kind of prototype analysis that I do. It's handy to just use off the shelf public repository data for that, and then the Kentik platform as well, brings in something really unique, of being able to bring traffic data into the story so then and we can get into it, but I think it adds a lot of value to understand right? George Michaelson 3:51 But using route views means that when you're collaborating with people outside that framework, you actually have a public assertion of state that you can point to. So for publication and reproduction using route views means other people can see broadly the same things you're saying. Doug Madory 4:05 It's one of the things I like about BGP analysis and the work I've been doing for whatever the last 15 years, is that a lot of it is eminently reproducible and verifiable, and so if you just give me the timestamps and the prefixes, someone else can go and find the same things. George Michaelson 4:20 So here we are production side, signing things about what we want the world to see in BGP. And you were talking about this a year ago, and here we are in 2025 so the obvious question to ask is, Have things changed? Has something happened in the space that's interesting as a dynamic? Doug Madory 4:37 Yeah, there's been a few things. Maybe the biggest thing is the milestone of crossing the 50% threshold, maybe perhaps an arbitrary threshold here of routes in the v4 global routing table, George Michaelson 4:47 50% because we're sitting somewhere around seven to 800,000 prefixes are being asserted. Aren't we [Doug: Yeah]. So there isn't necessarily a direct volume to the number of prefix announcements, but nonetheless. Half of the globally visible routing space is now protected by the assertion side. Doug Madory 5:05 Yeah, the way I would put it would be that it's eligible for for the protection, eligible because you also need, you know, the part two, but creating your ROA, you made it eligible to be protected by ROV. George Michaelson 5:15 So what do you think has driven this pace of uptake? Because if we compare this to IPV six. This is really a very rapid adoption curve. Doug Madory 5:24 Yeah. I mean, there's a lot of things that go into it. As a lot of engineers around the world that deserve credit for working on it. I give Job Snijders a lot of credit for being an evangelist around the world. But I think probably, if you were to look at the stats of when things change, it's around 2019, and 2020, when we had a handful of tier one operators begin rejecting invalids, and I think it became real at that point. George Michaelson 5:45 So this, actually, you could almost say it was a chicken and egg problem, right whats the point invalidating if nobody's signing. What's the point in signing if nobody's validating? Doug Madory 5:55 Yeah, so it needed a kind of a kick in the pants. And I think having the tier ones rejecting invalids made it real. And now, if you were to create a ROA, there would be a lot of there's a lot of downstream impact from tier ones. And if there is a, you know, a hijack that shows up that's inconsistent with your ROA, then those a bunch of the Internet at that point would then, you know, suppress that bad route. And that's the system working. George Michaelson 6:21 So we've spoken with Job about some of the mechanistic dynamics of this information space. He's been looking at collecting the data of the state of validation, and he's observed that there's been a really quite massive increase in the volume of data that people are now having to ship. So it's kind of the necessary consequence. We've hit 50% we've also massively increased the fetch burden to read all the data, haven't we? Doug Madory 6:46 Yes, and the cryptographic checking and stuff. So there's a in his you know, Job put it something out at the end of the year, looking at the growth in 2024 and one metric he's tracking is wall clock time of just the time it takes to update all the data, cryptographically check it, and now it's increased, and so it's something we have to kind of keep an eye on. George Michaelson 6:46 Well, there's no such thing as a free lunch in the Internet. If we want to make this milestone outcome of half the network being protected, we have to wear the consequence. There's a lot more volume of data being seen. So moving from production to consumption side, you are also tracking rates of ROV. Can you talk a little bit about how you're able to do that? Doug Madory 7:28 Yeah? So I think that one's a lot trickier. You know, like the ROAs are public and for everyone to see, George Michaelson 7:33 you just fetch them. Yeah, you do work on them against BGP Doug Madory 7:36 I guess, before we move off of that, I would just say that the bit of analysis I've been adding in the last couple of years is to bring our traffic data to bear. And I guess, as a BGP guy, I love having traffic data, because, you know, a BGP route represents a potential path, but it could also be no packets. George Michaelson 7:53 So Google might announce their awesome service with one prefix, and it syncs 50% of the visible network and George's ISP, for various reasons, might disaggregate and announce 100 prefixes and nobody's going there. So your size of information in BGP state or in ROA state is not actually a direct function of how important for one of a better word your route is, but your traffic information means you can do a ranking based on volume. Doug Madory 8:25 Yeah. So that's where I was thinking again. lets see, about three years ago, Job and I were shouting out, I wonder if we could, instead of just counting routes or counting IPs, or we get percentage of IPS like, some of these are not gonna carry any traffic, and some of you are gonna be gargantuan. And so I've got this unique data set of this aggregate NetFlow corpus we can I can query, George Michaelson 8:45 for those who don't know, NetFlow is a standardized technique for taking information out of routers and switches and providing a feed of information like packet sizes, packet counts, drop rates, source and destination addresses. You can construct a huge amount of information from a NetFlow feed, Doug Madory 9:03 yeah. So it's a source, destination, IP, port and bite size. There's a few standard things that come with NetFlow and NetFlow record. George Michaelson 9:11 So you're making an aggregation that for the NetFlow resources, looks at the source and destination and asserts, well, was this covered by a ROA? Is this capable of being a protected net? Doug Madory 9:22 Yeah. Upon ingest one of our customers routers, creates a NetFlow record. It gets annotated with a lot of things from the perspective of the router, and one of those is a RPKI evaluation of the destination IP and then, but that's super helpful for me. Originally, that was created to answer the question of, if I were to start rejecting invalids, how much traffic would I lose, or what would I lose? [George: Yeah], because that was a concern a number of years ago. I think most of those concerns have been put to put the rest. And typically the answer is, it's very little, and it's not a thing of consequence. So you can go ahead and do it, but now I'm repurposing that function to say, well, what if I took all this traffic across all of our customers, and said, like, how much is going to. To these different getting evaluated these different states. So we have like RPKI valid traffic, which is a, you know, packets that are going to routes that are that would be evaluated as valid in RPKI evaluation. Sort of valid traffic, invalid traffic, unknown or not found traffic. George Michaelson 10:15 And in some ways, the primary measure here is really the unknown valid, because that's the adoption measure. Yeah, invalid is the magic. You don't want to find it, but we all know it's there. That thing there just shouldn't be there. Doug Madory 10:28 There's a bunch of persistently, you know, invalid stuff due to misconfigurations. [George: Yeah], such things will probably always be with us to some extent. But you're right. Most of it is the fight between unknown and valid and which is on top. George Michaelson 10:40 So this 50% measure, this is coming out of NIST. They're providing this kind of sense, and it's not just in the States. We're talking worldwide, Doug Madory 10:49 yeah. So that's a global thing, and I don't think we mentioned it beginning. But so the milestone last year was before v6 was actually the year prior. So v6 was crossed the 50% Yeah, it did. But you think about it, it doesn't have the legacy stuff, you know, like, George Michaelson 11:06 it's a simpler configuration for most ISPs. It's one slash 32 right? Doug Madory 11:10 You're not going to have just ancient stuff in the v6 world. It's all relatively new. And so I think that makes it a little more nimble when it comes to something like this, George Michaelson 11:18 right? So 50% bit of a milestone that's on the production side over on the consumption side. ROV, how do you measure that people are validating? Doug Madory 11:29 Yeah, so there's been a few different approaches people have taken to this, and each has kind of pros and cons. So trying to it is really difficult to remotely try to accurately guess for any given AS is it rejecting invalids or not? So Cloudflare has these intentional invalids that they send out for this purpose, for measurement purposes, George Michaelson 11:49 they're beaconing. Doug Madory 11:50 Yeah, they're basically beaconing. And then they have a little test that they'll run if you can answer it. Well, then you you didn't reject the valid, the valid. George Michaelson 11:57 So they're creating a sawtooth regular state of change in the network. There are some interesting dynamics here, because the cycle time to hold onto information has one lifetime, and the cycle time to refresh information has another lifetime. So I could imagine that the rate of seeing a new thing happen, the rising edge part of is it covered by a ROA that actually might be quite different to the declining edge the ROA has gone away, and I think the pace here might be interesting. But the primary thing is you can use the beacon to understand if somebody does or does not see a route that you know is valid or invalid. Doug Madory 12:35 You can test the one again. I think that's a very clever technique. I guess the one thing that I would say is that is probably good. That's probably, it's a, probably a very good way to just arbitrarily choose, you know, get so I think they arrived. This is from analysis couple years ago that maybe 6% of ASs is rejected values or something like that is a very low number. I'm not sure what the correct low number. I'm ready to believe this something small. It's gonna, we're not gonna have a [George: it felt plausible] It's fine. I guess it does, like under count the impact of what RV is doing where, just like the routes, not every route is equal, not every AS is equal. We're talking about these tier ones that are rejecting invalids have a huge outside influence on the Internet, whereas some sub network somewhere is not and in that metric they're created, they're kind of counted as equal in that 6% George Michaelson 13:22 Well, there's also this component that if a tier one that is central to routing surface for an entire region, I mean, I don't think they are. But if we imagine France Telecom, for some reason, provided a massive amount of tier one service delivery in a footprint like Asia, if they chose to adopt ROV, all of their downstreams that are doing BGP with them, in effect, get the benefit of ROV being applied in that central point, so they're not necessarily of themselves actually doing ROV. And unpicking that distinction, that kind of needs a lot of different points of view in the network, doesn't it? Doug Madory 14:00 Yes, then my I guess the approach, where I'm going with this is, I was like, Well, I don't know if I can come up with a better technique than that, but what I could do is look at it from a macro standpoint. Again, going back to the route views data, if we were to take the whole routing table, v6 before and evaluate all the prefixes, and then look at how many of the of their vantage points can see the routes, you'll see a separation in the populations of those that are invalid are seen by much fewer vantage points. It's quite distinct from everything else, the unknown and the valid. George Michaelson 14:30 So Geoff Huston has a model that is purely his own. It's arbitrary, but he has constructed a sense of what he thinks is the default free zone by compiling about 10, maybe discrete points of BGP into an aggregate whole. It sounds a little bit like you're taking 10 or some number of points in route views, constructing an aggregated view, and then looking at the difference of individual route views collected measures against that central point. Point, because there isn't really a single ground truth of what the default state is. Is correct. It's an aggregation over all the different views, Doug Madory 15:08 yeah. So I'm looking at basically all the full tables. So I'm like, so it's like, 400 [George: 400]. Yeah, I think it's what you need to do. What you ended up having is when you plot this out and the x axis number of peers that see a given route, starting with zero or one on up to 400 you end up with a big peak around 400 where that's the for those, that peak is all the globally routed routes that everybody sees. George Michaelson 15:31 So it was a fairly strong signal to you. That's where you need to be to say this is the globally visible state that matters. And now I can start looking at individual points saying, Do you see these things? What do you not see? It's like the Venn diagram intersection game. You You're finding the things that aren't there. And have you been doing this enough that you can get a rate of change in this? Doug Madory 15:52 Yeah. So we, I guess we first did this in 2022, and it was, again, it was very clear I had already kind of started encountering this anyway. It was if, a route is invalid, it just doesn't propagate very far. How far it propagates really depends on who are the upstreams, and it becomes a complex it's very hard to predict, but at very least it's gonna be, you'd be really hard pressed for starting anywhere on the Internet to get half a propagation of half the Internet within a valid route. It just can't go very far. George Michaelson 16:19 You know, the thing is, we couch this in a negative language, oh, no, the invalid rope won't propagate far. But actually, that's a positive outcome. Doug Madory 16:28 Is the system working? George Michaelson 16:29 So yes. So this means we have limited the surface of visibility of blatantly invalid routes. When people create rowers, they don't propagate right? Doug Madory 16:39 We want to suppress problematic routes, we have a system. The system is automatically identifying which ones are problematic that would be disruptive, and they don't go very far. And that's what we want to happen. Now you asked about a rate of change, saying that we first ran this in 2022 again. We ran again last year in a couple different ways, and we can see that the propagation appears to be slowly decreasing, while those that are valid are, I guess, increasing with the size of the route views, Database, Data growing. George Michaelson 17:12 So one of them has a natural rate of growth. It kind of tracks growth in the overall route count, and the other has a rate of decline, if you like, distance carried and number of associated AS BGP speakers who are affected, that is tending to shrink, [Doug: yes], and it's kind of going to arrive at a place where it's likely, if you have a whoopsie and you assert something that is blatantly invalid, it might only be your immediate adjacencies who have to deal with that. Doug Madory 17:40 Yeah, we have a, there's been a handful of stories. Again, as a storyteller in the space, I'm sure you can relate. It's, it's hard to tell the stories of the things that didn't happen, [George: yeah], thing that didn't go wrong. Like, who cares? Like, what didn't go wrong? Like, why do I worry about it? But, you know, I mentioned in the talk this week was so in September, there was .. in Brazil. There was the government of Brazil gave an order to block Twitter. I'm not sure what was happening at the time. George Michaelson 18:03 Yeah, politics. We don't do politics. Doug Madory 18:07 So a couple of Brazilian providers took it upon themselves to use BGP to Black Hole Twitter. The same thing that took place in Myanmar in 2021 and Russia in 2022 at least between the time of Myanmar and Russia Twitter now X created ROAs for the majority of their routes, the most important ones, and in this case, in September, one of those providers leaked the Twitter routes, but they only went to the immediate peers, and not the transit providers of the Brazilian provider, and there was no outage outside of Brazil. As a result, I would it's hard to know definitively what was the mechanism that prevented that. George Michaelson 18:44 The ground truth for people to understand the scale of risk, here would be the Pakistan YouTube incident that was pre RPKI, [George: sure]. And so that had global reach, because there was absolutely no boundary on the more specifics they asserted on the wrong side of their boundary. And so you're kind of saying the good news story here is they did a thing that had potential to be globally visible, and it wasn't because people said, that's just not true. And so it was asserted as an invalid and it was dropped as an assertion into global routing, really quite close. Doug Madory 19:17 So I guess I don't know how many people even know that that took place. It really could be less than 10. I don't even know the provider knows that the leak. I mean, I came across it, I found evidence of it in a couple places, and also could prove that there was no disruption. George Michaelson 19:31 But again, a leak that doesn't have scale, it's not good, but at least the damage is incredibly small. Doug Madory 19:38 Yeah. So I think it's important, again, as someone who speaks about this, to point out these I call that a success. Too bad for the folks in Brazil. We can't save them for themselves, but this happens around the world in one form or another, on a fairly regular basis, and if we can just contain these bad routes within the country that they came from, that's the best we can hope for. I think George Michaelson 19:58 so. Have you been able to take. Track ASPA. Do you see any visibility of ASPA in the system? Doug Madory 20:04 I haven't, haven't really built good tooling to try to dig into that, but it's fairly low. I think right now, another thing that helps with our ROV and the ROA creation was the ability, like, outside of a couple of outfits like Amazon or Job himself, like there's everybody's typically using the RIRs. George Michaelson 20:23 Yes, they're not generating these cryptographic asserts in their own machine. They're using centrally managed facilities to create the state. Doug Madory 20:30 I guess if someone's listening to this and they're thinking, this is a daunting thing, and they're concerned about, you know, like, I don't wanna have to learn cryptography or something. There's a workflow that's already built for you within your RIR, if you're an APNIC person, myAPNIC George Michaelson 20:44 goes in pretty much we you press one button, we do the rest. Doug Madory 20:47 So it's the relic creation can be very easy, George Michaelson 20:50 but ROV, ROV is a little more complicated, [Doug: correct] You have to think about actually running a system that is passive monitoring for some period of time. You might need to have a digital twin of your own routing surface, and do a lot of what if work, because this is a space where you have the potential to make things go wrong. Now, your accounts in Kentik that do the net flow. They're a really interesting tool here, because you're able to say to people, Look, if you did do this, this is the likely impact on what you're doing. You can help contain the fear side of the FUD problem. But you can't just turn ROV on with a one button press. Doug Madory 21:27 It is more involved. And there's, there are a lot of resources out there, but yeah, of the two steps, that's the harder one. George Michaelson 21:31 So you're seeing an increase in ROV, and we've seen a 50% line in ROA. Is there an equivalent 50% line in ROV. Is there a signature point that we could have here, or is it not really an applicable scaling? Doug Madory 21:46 That's a good question. I guess another milestone last year was Zio rejecting invalids. So of the reason why people could debate who are the tier ones, George Michaelson 21:56 undefined concept! Doug Madory 21:58 That's another topic, like politics here, but like of the of the traditional ones, in my mind, zyia was one of the last ones that was not rejecting invalids, and now that they are that's greatly beneficial. It probably helped move the numbers last year. They deserve credit for getting that done. And now we're down to again, of the of the traditional ones that are. In my mind, Telecom Italia is the last global backbone that is not rejecting invalids. And there would be great benefit if they the last, last big one. So I guess, you know, as far as a metric, if... George Michaelson 22:32 so, we could say all of the tier ones are done. We could, actually, we could move into a sphere of seeing it moved from the center out. I mean, APNIC engagement in this space is typically training in emerging Internet economies. So we've had increase in production side really quite easily in large economy like Bangladesh. And we've got phenomenal ROA uptake, [Doug: amazing isn't it]. It's 90% but the ROV side is just a lot harder, and it really is single digit percent of the AES surface in that economy is able to get into this space. And I do think this central model, where we start at the inside the core of the net and move out may actually achieve more outcome here. So, is this a linear trend? Is this what is the trend line here? I mean, is this a continuing trend? Doug Madory 23:21 Yeah, I guess for a while, for quite a while, like the ROA creation, if you look at the grass from this is almost uncanny, how straight the line is, like, like, it just seemed like an unnatural straight line heading, you know, up into the right. And the whole time I'm looking at I'm like, All right, at some point we're gonna start running into, you know, Legacy address space that has some baggage, and we can't get the ROAS created without George Michaelson 23:45 there's no natural law here. There's no physics component here. This is a logistic issue, and logistic curves have a very weird shape. There's slowness at the start, then rapid uptake, then a kind of boring midlife and then you hit midlife crisis point, and things kind of change. [Doug: Yeah]. So are we seeing science Doug Madory 24:01 change? Yeah. So I think I know with with Jobs analysis for the RPi database for 2024 we weren't seeing a lot of movement in v6 so a lot of the growth that had they v6 was probably out in front of v4 yeah, as I mentioned. And and then that seems to have slowed for v4 ROAs are still getting created from the traffic standpoint that I've been checking when I started looking at this back in I think, I did my first NANOG presentation on this topic in February 2022 we were at 1/3 of routes with ROAs. And the big insight was in the traffic standpoint, we were over 50% in bits per second George Michaelson 24:05 and meaningful traffic. Doug Madory 24:08 Yeah, that's really going maybe, and the majority, and that's mostly due to eyeball networks and content providers, the RPKI deployments, pushing more traffic than their route counts would, would lead you to believe. And then last year, we brought that up to now we're majority, or over 50% of the routes and the traffic level by late summer in the fall was like 73% packets per second, so we're approaching three quarters. However, when I've checked it recently, it's at 73% so we've been kind of stuck. George Michaelson 25:10 So we may have plateaued. Doug Madory 25:11 We may have plateaued in just the percentage of traffic heading to routes with ROA. George Michaelson 25:18 And it could be we are in classic 8020 land, the closing that gap is going to incur a significant cost burden in outreach and in helping people and in just getting impetus to change, because we're down in the areas where people don't move of their own accord easily. Doug Madory 25:36 Yeah, there's some, you know, there's equipment issues. There all kinds of excuses are reasons why it's harder for a network to do this, or they Yeah, so I think the gains now in that space, they may be harder to be gained. George Michaelson 25:49 So Doug, if people were interested in this, is there a place you're publishing things about this, they can go look at? Doug Madory 25:55 Yes, there is. So I write quite a bit on the Kentik blog. And if you were to Google my name, Kentik and RPKI, you can see George Michaelson 26:04 Doug Madory, Kentik with a K Doug Madory 26:05 that's right, Kentik with a, k, k, e, n, t, i, K, and I look up RPKI, some of this I've written jointly with Job Snyder's, and then some of it done on my own. George Michaelson 26:15 But you're an occasional author on the APNIC blog as well, and we cover all services here. So I like this idea that you're actually having a consistent view of this over time. Do you think this might become something you're visiting again regularly? This could be popping up in NANOG as an annual State of the Nation report. Doug Madory 26:34 It could be, it could be, I definitely will be continuing to track the rates of adoption. And then, you know, in a year, we may, yeah, something new to George Michaelson 26:42 and kind of, there are going to be incidents of significance in this surface as well. So we could have the production story and the consumption story, but we also have the whoopsie story that's going to break out from time to time. And people love learning about these things. Doug Madory 26:56 It's how we learn. It's not their network. George Michaelson 26:58 I think the ASPA story is actually quite interesting. The fact is, we're kind of at Ground Zero. We're at the zero start, and so it's a rare opportunity to know we could start to track the reality of emergence when this finally hits the engineering train of the router software, the vendor router software. That's the point. We want to have the information Doug Madory 27:21 collected now that, as well as similar workflow that's makes it easy, as easy as it is for someone to create a ROA in right through their we need one button. We need the same thing for ASPA, some similar easy workflow, just to take the complexity out of it. For people, I think when that that's awesome. I mean, obviously you need the route the vendor support in the routers, George Michaelson 27:42 yeah, but mechanistically simple ways to produce it, and then adoption of consumption side, when it's seen as viable for use inside the routing, although it has a higher cost, doesn't it? This is not cheap to compute, Doug Madory 27:55 yeah, I guess. Compute, compute. What to just compare George Michaelson 27:58 Yeah, to do the path analysis against the chain of ASPA, because you've kind of, it's ASPA soup. You're constructing plausible paths out of this. So BGP, it was an absolute, concrete assertion. The nested chain of signatures simply are the Protected route ASPA they should be. Doug Madory 28:16 You've got to go through the AS path and figure out, Is there a value free violation in this George Michaelson 28:21 so it wouldn't surprise me if it turns out there's another adjunct processor, like RPKI, RTR, doing this work off router and providing a feed. I think we might see some complexity in the space. That's the story for the future date. So you might be checking back in on this one again. We might get you back on ping. Doug Madory 28:39 I'd love to come. George Michaelson 28:40 That would be great. Thanks, Doug, it's been really interesting. Thanks. If you've got a story or research to share here on ping, why not get in contact by email to ping@apnic.net or via the APNIC social media channels. Also remember the measurement@apnic.net mailing lists on orbit is there to discuss and share relevant collaborative opportunities, grants and funding opportunities, jobs and graduate placings, or to seek feedback from the community on your own measurement projects. Be sure to check out the APNIC website for all your resource and community needs until next time you