Geoff Huston 0:00 So let's be Paramount for a second, and someone connects to their streaming server and says, Hi, deliver me this podcast. And they kind of need to know, where is this user coming from? Where am I located to deliver this content, and am I able to deliver that content? Do I have the necessary permissions based on national licenses? So if I'm Paramount, I know where my data center is. I know where I am, but where are you? And this is where we get into this vexed question of geolocation. And I'm not even concerned what city or state or anything. I want to know what country you're in buy, well, what? buy your IP address George Michaelson 00:56 you're listening to ping, a podcast by APNIC discussing all things related to measuring the Internet. I'm your host, George Michaelson, this time I'm talking to Geoff Huston from APNIC labs in his regular monthly spot on ping. For over a decade, Geoff has been using Google advertising to place tests into the worldwide Internet. Up to 35 million samples per day are collected from users distributed throughout the global Internet. This measurement exercise relies on information about the approximate geo location of the device based on its Internet address in either IPV4 or IPV6. A number of techniques exist for compiling this information from the declaration made at the time of delegation of the addresses to the Regional Internet Registry through data collected by BGP speakers, such as content distribution networks, who have a massive interest in finding the closest source to any specific user request or the declarations made by address holders themselves, most of this data winds up in The various services like Maxmind who compile and publish geolocation data at varying levels of granularity and make it available. Recently, the massive increase in the use of starlink, a worldwide Low Earth Orbit satellite based Internet service, has placed strain on this geolocation process, and this has led to distortions in the data mapping Internet usage to the various economies and regions worldwide. Geoff has had to make some hard decisions to try and ameliorate things in his experiments. Geoff, welcome back to ping. What should we talk about today? Geoff Huston 2:44 Hi George, look. It's good to be here. The first question I want to ask you is, where are you? George Michaelson 2:49 I'm in Brisbane, mate. Geoff Huston 2:51 There you go, a geopolitical location. You know, you're somewhere on the surface of the planet, and as far as I can see, you're in the country called Australia, in a city called Brisbane. George Michaelson 3:02 Oh, you want the other one? I'm in leap, Tel. I'm in Leaptel land, Geoff Huston 3:07 You know, location is kind of interesting in all kinds of ways. And you know, most of us are aware of our location sometimes, you know, we're quite happy to proclaim it. Sometimes we're not so happy to proclaim it, George Michaelson 3:22 mainly when somebody like Netflix says, I'm sorry the content you requested is not available in your region at this time. Geoff Huston 3:31 Well, yes, it could be an IPR thing. It could be like the poor folk in Afghanistan at the moment, where there's some massive blocks on on Afghan traffic. And if you could tunnel your way out of the country somehow and pretend to be somewhere else, you might have a better chance of opening up a communication system. So, you know, geolocation is used for all kinds of reasons. Yep, George Michaelson 3:54 yep. And some of those reasons we clash into but a lot of those reasons actually work to our benefit, don't they? Geoff. I mean, the mechanisms you've discussed about guaranteeing that you get content from the nearest source, in packet delay terms, is fundamentally about geography, isn't it? Geoff Huston 4:13 Well, yes, trying to figure out what's the closest between source and sync gives you faster service and better service, but even sort of the social things, and you know, it's, it's, it's a very raw nerve in Australia this week, but that the emergency call systems. Hi, I'm in a deep deep Pickle, can you send around some emergency services, you know, for whatever reason, and they're expected in the call framework, Nat, to actually locate the caller and to have that on the screen of the responder so that they know where you are. Because sometimes, when folk are terribly panicked, they might not actually be able to describe their location, and so you know, networks need location for a whole bunch of different reasons. George Michaelson 4:56 In the case of the emergency services, if you take an overlay telephone service from an Australian provider, they are legally required to inform you that your location service mapping for emergency simply may not work adequately because they can't guarantee, given that you're using an IP address, they can't guarantee that they can reliably detect your location. And so they say up front, fine, we're happy to give you voice over IP over this tunnel, but please bear in mind you no longer have a guarantee of location if you dial triple zero. Geoff Huston 5:28 Well, the reason why they have to say that is actually embedded in the old view of the fixed line telephone system. Mobiles went and changed everything. But, you know the fixed line system, because, you know, my number started with plus 61, I'm in Australia. Then it had the next two numbers. Three numbers were Oh, six, two, which actually located me into Canberra, [George: yeah] And then, from then on, the numbers were more random than that, and they were just blocks of numbers, but basically the address plan and the telephone system was geographic, and it was the telephone number itself that gave very solid clues as to where you might be on in the world. A whole bunch of reasons, but the basic reason was technically, it was simpler. George Michaelson 6:21 It was really quite a clever design, when you think about it, because it was not only simpler, it was equitable between economies. I mean, there are a couple of biggies that managed to get the magic behavior one digit up the front plus one determines that you are in the North American continent. But pretty much every other economy worldwide is identified by a two digit pair with a small number identified by three digits. And the second thing about it is it was extensible. It's not determined up front that there can only be five or six or seven or eight digits in this number. You can have 15 or 20 digits. So as you grow the size of your market, you can grow the length of the numbers that you put into one of these addresses. It had some upside benefits. Geoff Huston 7:08 So in this system, which went by the rather bizarre name e dot 164 don't have a clue why every country was assigned a two digit code, Australia, plus six, one. And when I say country, I don't mean it, you're right. The North American Numbering Plan, which included the US, Canada, Canada and some Caribbean islands, took all of plus one because they were there first, I guess, you know, and every other country got got a number prefix, which raises the fascinating question, what's a country? Wow, don't we know what a country is? George Michaelson 7:43 Oh, if only we knew Geoff that is such a hot political topic, isn't it? Geoff Huston 7:48 It is, and it's got all kinds of overtones. And there's this weird area of the International Organization for Standardization, ISO, hang on, that's the wrong. The wrong capital letters. It should be IOS. No, no, no, it's French. So the French, the French title for the International Organization for Standardization, has a standard that goes by the numeric code, 3166, ISO 3166, and it has a maintenance agency. What do you mean? Well, that's what it's called. Its job is to maintain the ISO 3166, standard. [George: Yeah], now that has 15 experts. What do they do? George Michaelson 8:31 They argue with each other around the table like every other collection of experts. Geoff Huston 8:38 Oh, totally. Is this a country? There's a benighted bunch of rocks inhabited by a large number of penguins, I think, Heard and Macquarie Heard and McDonald islands somewhere in the Antarctic Ocean down there in the south, it's a country, well, it's a territory, but it's a country. Well, it's a territory. Well, ISO, 3166, Bestows on every recognized country recognized by the maintenance agency a two letter code, [George: yeah] and so HM, is a two letter country code for this bunch of rocks, George Michaelson 9:15 yep. Now 15 member committee. I'm coming at you with an international organization based in the continent of Africa that coordinate fishing, the African fishing coordination agency. We'd like a two letter code, please. What have you got? Needs to start with A by the way, want it to start with A? [Geoff: Well], I know, let's get AP. Geoff Huston 9:39 In theory, until people like that started knocking down the door. It was pretty simple. If you're a country, you had a two letter code, a three letter code and a three digit number code. And if you weren't a country, you didn't but you know, George Michaelson 9:54 Hong Kong HK now is Hong Kong a country Geoff, Geoff Huston 10:00 a bunch of political decisions, even the former Yugoslavian, what was it called Confederation? It was a, it was a bunch of component states that then split up George Michaelson 10:10 the former Yugoslavian Republic of Macedonia, d, FYROM, Geoff Huston 10:14 right? All of those. So they kind of started going, they put up their heads. I want a two letter code. The USSR had a code, but now it doesn't. So there's this whole thing about what's a country? When does a country stop being a country? When do you retire the code? And oddly enough, this is all tied into geolocation, because normally it's it's like I could give you the latitude and longitude coordinates of where I am on the surface of the planet, but quite frankly, unless I have a remarkably good map that details every geopolitical border in terms of latitudes and longitudes, I'd be kind of stuffed. George Michaelson 10:54 I think, I think in some ways, this is the collision of technology drivers to finding out where you are in the way machines might think about this and human behavior to finding out where you are, and the sense that, although we might struggle on the margins, defining what is or is not an economy and what should or should not have an economy code. The fact remains, we know it when we see it. We actually understand the idea here, as individuals, as humans. This is a pretty good way of aggregating certain behaviors. I speak Romanian fine. Then you're going to get the .ru code to be a Romanian, except for it's not RU, but you know what I mean, Geoff Huston 11:32 RO, and don't quiz me. I know far too many of them, and that's a sad fact. But the issue is, I think that that country delineation is actually political. What laws apply to you now? You talked about Netflix earlier, and yes, the intellectual property area is a big user of this kind of data, because they sell the rights to redistribute based on politics, based on which country, which operator in which country has a license to redistribute that property. George Michaelson 12:04 And there's probably a quite close match between economies, countries and political jurisdictions and legal jurisdictions, the ability to constrain a contract, the ability to put terms on the table, the ability to be a registered business entity and form an agreement with Netflix or with the intellectual property holder. Paramount, for instance, are going to go out with their lawyers and deal with lawyers in other economies. Lawyers don't float around on ships in the international region, capturing business. They are located in a country. They deal with the laws of a country. Geoff Huston 12:41 So let's be Paramount for a second, and someone connects to their streaming server and says, Hi, deliver me this podcast. And they kind of need to know, where is this user coming from? Where am I located to deliver this content, and am I able to deliver that content, do I have the necessary permissions based on national licenses? So if I'm Paramount, I know where my data center is. I know where I am, but where are you? And this is where we get into this vexed question of geolocation. And I'm not even concerned what city or state or anything. I want to know what country you're in. By, well, what? [George: Yeah], by your IP address. George Michaelson 13:26 Hang on red light, warning here, danger signal. Geoff Huston 13:30 Well, things get a little hard, because you can sort of say, well, why don't you ask the. Internet service provider, and they'll tell you where this address is located, you've got to go, Oh, wow. I don't think we're going to live long enough to do this. No one keeps consistent records. No one's even obliged, in sort of the International Space, to offer answers to random questions about location. Who do you ask? George Michaelson 13:55 I've had very direct experience of this Geoff. The provision of telephone services in Latin America is actually quite closely allied with an international combine of telephony providers based in Spain. Telefonica has a massive amount of sway and their address pools that they acquire from any region as they see fit, wind up being deployed where it makes most economic sense to them. So you can have an address that in registry terms is recorded as being assigned to Chile popping up in mobile Internet in Uruguay quite comfortably, because that company doesn't feel a strong need to respect these boundaries. Geoff Huston 14:34 You just walked over a trip wire, which is the next thing I actually want to talk about, because trying to sort of map IP addresses to countries. The kind of naive answer, the go to answer, is to go back to the Regional Internet Registry that has a record of that address assignment, and if you follow the public published information and target the threads hard enough, you can find a block of addresses and an organization or entity and a country code, [George: yep]. And it sort of infers that these addresses have been assigned or allocated. Who cares to this entity, who is located in this country? George Michaelson 15:20 Yeah. I think we can all guess what's coming next Geoff, because where an entity is located and where an address is used.... Geoff Huston 15:30 Ah, you see that that's kind of the disconnect, because from the regional Internet registries kind of perspective, this is an administrative address. Where does this entity have its headquarters, its domicile? Where do I send the notices of demand to pay up or whatever? You know, where are the holders? NOT that's a big not. Where are the addresses? Because if you run a network across a number of countries, and you, as you were saying, move addresses around. I remember the Swedes allocated some of their addresses, one Swedish telco to an operator in Kazakhstan. And it really did look like an anomaly that what were addresses assigned to a Swedish entity were being used in what was pretty clearly somewhere else on the planet and what looked strongly like Kazakhstan, George Michaelson 16:20 Yea. And so if you, if you think about this in terms of the behaviors when we entered this game back in the late 80s, early 90s, 8020 rule, or maybe even better, most entities operated within the legal boundaries of the economy. They were in, by no means all, but most. And so giving out addresses to entities registered in a given economy was, for most purposes, a broad brush indicator where the addresses were likely to be used. The problem would be that that was never 100% true, and it is now a long, long way from even 80/20 true is it? It's just massively disconnected Geoff Huston 16:59 the Regional Internet Registries talk about the "where" of the who owns the addresses, who's the entity that, if you will, was given these addresses, who's the upstream ISP? Where are they located? But the question, the, you know, Paramount wanted to ask about where is this user located, is actually a completely different question. [George: Yeah], where did you assign those addresses? And the RIRS have no mechanism to report that, and indeed, don't record it in their registry. [George: No], I give you a pool of a million addresses, and you assign 50,000 to Country A and use 60,000 in Country B, etc, and then on Wednesday, you go and change everything. You don't tell anyone. There's no reporting mechanism. George Michaelson 17:44 So there's actually a moment buried in this, because there are a significant number of people who don't align with that idea, who actually think the place that you say this is the distribution of my addresses, is inside the registry model, using registry public data records to make changes to this. And at one level, if that was an enforced choice, and was an understanding amongst the ISPs, they did this and they manage this, that could work. The problem is it's not actually a real choice that can. Really be implemented that is implemented consistently worldwide by all players, even within the APNIC region, it's not implemented consistently. Geoff Huston 18:24 And so this whole country code, you know, if you actually look inside the registry maintained by the ripe network Coordination Center, the RIPE NCC, you will find a whole bunch of allocations to this mythical country that goes with the code EU. That's not a country. It's not and it's kind of even in Asia Pacific, I look at the APNIC, and I find a couple with the bizarre country code of AP, that's not a country George Michaelson 18:52 the African fisheries forum!. Geoff Huston 18:54 So even the original Internet registries are not consistent in if you will, tagging registrations with the country where the entity is actually domiciled. And secondly, the units are wrong. [George: Yes], I give you a million addresses, and you put 10,000 here, 50,000 there, etc. The registry doesn't cope with that. It just simply says, I gave you this block of addresses full stop, so you're kind of looking in the wrong place. [George: Yeah], it's not a good place to start. So where should you look? There's no publicly funded public service location system, not even within countries, because the addressing system was kind of an imposition that was supranational, not domestic. There's no equivalent to an Internet numbering authority in most countries. They do in some places, but not in all. George Michaelson 19:51 I think it has the feeling of a classic externality Geoff, because the people who are looking at this saying we need one are the Paramounts and the Disneys and the legal jurisdictional boundary operators, firewall operators, national policy operators saying we need to know. We need to know. We need to know. The ISPs are looking at it going well for technical efficiency reasons. We can see you want to know how to route traffic to a CDN, and we understand that could work to the customer's benefit. But we, as ISPs, we don't actually need this. We know where the addresses are, [Geoff: right, so] there's nothing in it for us. So why should we acquire the burden of doing all this maintenance work when it's your problem? Geoff Huston 20:32 So of course, the other the other customer is law enforcement. But I think the argument still holds, why should I go to the burden of maintaining all this and secondly, I maintain it using my standards and my internal Excel spreadsheet. I'm not going to make that public information George Michaelson 20:49 spreadsheets run the world Geoff Huston 20:51 afraid they do. I'm not going to publish it. It's commercially sensitive. It's kind of, here's a list of my customers come and come and come and steal them. [George: Yeah], you know. So there's very little motivation for this information to actually may be managed in such a way. And as I said at the outset, we do not use a geographic address plan. So there is no template for where things are, none. And of course, in the deregulated world of the Internet, a number of commercial operators have come in to plug that gap. The big content networks spend a lot of time working through geolocation, but it's not what you think it is. If you look at the fiber map of the world, the connectivity map, and you look at the normal map of the world, it might well be that the network path from A to B is quite long, whereas the geographic path is kind of over that hill just to the left there. And so for many folk, the location is relative to the network topology, not relative to the geo you know, the physical landscape, the sort of spatial geometry of the surface of the Earth, George Michaelson 21:59 right? It becomes a lot closer to a mapping to BGP and the originator in BGP. Then it comes to a static declaration of intent to location, Geoff Huston 22:09 right? In routing terms, where distance is really kind of less relevant, you get some odd adjacencies that you'd never think of in many systems. Australia is geographically adjacent to Japan and to the US physically, their oceans away and so on. So where do we go from here? Well, a number of folk did this internally, and some others did it as an exercise where they could sell the data. George Michaelson 22:36 Yeah, and sell is a very important word here. This has strong commercial value. Geoff Huston 22:43 Oh, it has commercial opportunity. I'm not sure anyone's a millionaire. And interestingly, the fine, granular. Parity of data down to the city is normally sold for some money, but IP addresses to country mappings, that's pretty coarse, and a number of folk release that information and maintain it as generally available without cost, [George: yeah] And being the cheap researchers that we are, if we can avoid spending money, that's a good thing. And these days, in APNIC, we rely on two providers. We think they're both reasonable, Maxmind and a more recent one, IPinfo.io, and both of them, both of them have interfaces that allow us to either put an IP address into their system and get back a country code, or take the entire mapping database and do that work internally? [George: Yeah], here's an IP address. Tell me. Tell me, what country does it look likely they're not perfect. [George: No], they get fooled by things like VPNs, etc. But I would say in general, we've had a look a few times and used a number of sources, and about 2% of the entries are a bit weird, yeah, but the other 90 98% are okay. George Michaelson 22:53 I've played a little bit in this space over the decades, and it is a very interesting intersection of a belief in what a public benefit would be, what's commercially viable and acceptable as an error rate and what people, for commercial and confidence reasons, keep to themselves. The large CDNs have a very, very good understanding of the efficient distribution of addresses to their data centers. And when you ask them directly, could that become part of a public data set? They have a conversation internally, and they come back and say, Look, we get that you're heading to a public benefit, but it actually gives us a commercial advantage in the marketplace right now. So it's not that we can't see why you'd want this. It's that it actually doesn't help us to construct this outcome, given there's a competitive market in CDN routing, you go to people like Maxmind and say, hi, we think you have some errors in your data, and we'd like to help you correct it, and they quite correctly, say, Look, I know who you are. I've met you at conferences. I know you work for a public interest agency. Why should I believe you when you tell me you know better than the guy who told me the addresses and said he was in charge of routing them, they actually have concerns about people trying to help them fix their data, because not everyone is a good operator. Some people want to deliberately change the way packets route. Geoff Huston 25:13 Many years ago, some Czechs got in touch with me at a conference and said, You seem to be recording a bunch of networks inside Czechia, the Czech Republic, that ARIN. What do you mean? He brings out the list that one, I think you'll find that's further over to the to the east, head into the general direction of Russia and that, and that. And so a number of providers were trying to, if you will, game the system and transmogrify the origin country of their addresses to a different country in order to get around some restrictions, whatever. And so you can't necessarily believe that every piece of information you're told about where where a network might be, is correct. There is an interest in lying. George Michaelson 25:59 We were contacted as the registries by a hotel organization in the Caribbean, saying that they were very uncomfortable that one hotel in region was able to show American daytime TV and none of the other hotels could, and they felt we were disturbing the market, and we really couldn't understand what the problem was. And then we clocked into the fact that one hotel had addresses that showed as part of the North American Numbering Plan, and the other Caribbean hotels had addresses that showed as part of the Latin American distribution framework. And we perturbed the market. Geoff Huston 26:33 When you say, we, I would, I would actually say the folk who were implementing this restriction system had fallen for the trap, [George: oh, sure], in believing that the RIR data talked about where, not who. George Michaelson 26:47 Oh, absolutely, it was a misattribution. the entry condition, we were in some way. Responsible Geoff Huston 26:54 yes, the whole thing was based on a misinterpretation of the data. [George: Yeah], so I want to move on from that into what we've been doing for some years and a rather bizarre misinterpretation. We use this data, yeah, and we actually combine it with our advertising based measurement system. George Michaelson 27:12 We here being APNIC labs, APNIC labs, Geoff Huston 27:15 APNIC labs. So we spread out about 35 million ads a day across most of the planet, most of the planet. If you've got a browser and Google knows you know your existence, Google is probably going to try and serve you ads, whether it's on your mobile phone, on an app or on a browser. You know you will get served ads and our ad is part of this. So most folk receive ads from us, from everyone else. Now we're an advertiser. We get to see the IP addresses of where the ad is delivered. Interesting and with a few assumptions, and one of the major one being that, if in a country, it might have three ISPs, each of which have about a third of market share, then you know, to a first degree of approximation, we actually see an ad frequency distribution which equals the same 1/3 of the ads go to Provider A, users in Provider A, 1/3 go to users in Provider B, and the remaining third to users in Provider C, give or take. And oddly enough, we can use this system to make a pretty crude assumption about market share. [George: Yeah], you see most markets for the Internet, and it's predominantly dominated by mobile and the issues around spectrum tend to say that most countries have three majors. Yeah, three big providers. Sometimes it might only be two. Sometimes it's a monopoly, but it's rarely four, rarely five. There's not enough spectrum space to go to the highly populated areas of any country and put up five different spectrum allegations and expect it to work. [George: Yeah] So in general, this kind of system is it kind of gives us reassuring answers that our intuition is kind of right. George Michaelson 29:03 I've gone out into the community talking about the ads many years ago, and said to people in the room, here's a list of the top five, top 10 entities based on the ads relative weighting. Do you guys have a major problem with that distribution? And I've had people say I would flip position three and four, or saying I think one and two are a slightly different scale but broad brush. They all said, yeah, that's pretty accurate within the limits that the stuff that says Geoff Huston 29:32 within some limits, and I would kind of quantify that as about the numbers in terms of users, might be plus or minus 15% you kind of go, that's a bit weird. But the answer is no, because most internal systems. Most domestic environments have small number of biggies, most of the users, and then long tail of more specialized [George: Yeah]. So okay, we've been doing this now for some years and publishing this data, George Michaelson 29:56 It's over a decade Geoff, it's heading up to 15 years now, Geoff Huston 30:00 you know, oh God, don't want to think about it. So, you know, we get a few complaints going. We should be number two, and you're ranking it as number three. No, you don't understand. But we got a report a week or so ago about Yemen, and Yemen was kind of curious. Yemen. Yemen has around about 10 million people, give or take George Michaelson 30:19 For people who don't have good geography. Yemen is a country at the bottom of the Arabian Peninsula, nestled between Africa and the land mass continent of Asia. So it's an economy right in the middle of the Middle East Geoff Huston 30:35 Adjacent to the Red Sea. A whole bunch of shipping goes past Aden on their way to or from the Suez Canal. And the stats using our system were kind of weird. It has one big provider, YemenNet, that is presumably operates fixed lines and mobiles and probably predominantly mobile. And there's a second more specialized provider, which I guess is in the corner of the country dominated by the city of Aden -adennet and two others. One is Cloudflare net. Now Cloudflare operate the egress gateway for Apple's private data relay. They operate their own gateway for warp, their VPN product, and around about a quarter of a percent of users in Yemen are using Cloudflare. They're kind of but that's what the ad showed you, kind of go, yeah, that's about right. A small amount of corporate usage using VPNs. George Michaelson 31:32 We we see Cloudflare popping up in a lot of economies. And to their credit, they actually have tried to be fairly careful about their address distribution methods. They provide an overlay map to give you broad brush indication. Geoff Huston 31:44 So to Apple, you know, that's kind of consistent. So what's odd? [George: Yeah], out of these 10 million people and 3.4 million users, 60% of the ads go to or come from IP addresses that are announced by AS, 145930, so what? George Michaelson 32:02 I don't know the AS map. Geoff Huston 32:04 starlink, wow. Are you saying that two thirds of the country, and you know, two thirds of the country are using starlink, and it's kind of [George: Wow]. I'm not sure that's true. George Michaelson 32:14 That would be huge. That would be massive. Geoff. Geoff Huston 32:17 It's not true. They can't afford it. The GDP per capita is not high enough. And as far as we can see, no That measurement is dubious. George Michaelson 32:27 Yeah, this is an economy of 40 million people, and if they've got 10 million people online, they're by no means saturated market. They're not a rich economy per capita. There's no way that people could afford to buy the most expensive Geoff Huston 32:42 6 million of those 10 million, presumably, if our assumption about ads is correct, use starlink and it's kind of nah. So [George: no] a number of kind of theories around there. But let's think again about Yemen's location. How many ships are waiting to go through the Suez Canal? George Michaelson 33:02 Oh, wow. There's visualizations Geoff. You can see them from space. Geoff Huston 33:06 Well, you can also use a whole bunch of, you know, a bit like plane tracking, and there's ship tracking apps as well. 60 ships a day currently use the Suez Canal, 60 [George: 60], and it takes a few days to transit through the Red Sea. Now they could be still using Inmarsat for their communications purposes. Yeah, right. You're laughing too. So am I? [George: Yeah, no], really expensive. Doesn't work. Geostationary satellites go away. No, they're gone. All of them use starlink. It's so cheap and so fast. So I'm a ship and I'm in the Red Sea. Where is my IP address geo located to now starlink and nice people in so far as actually publish an IP address to country map, [George: yeah]. And what we find is that each each day, the users of someone around there map into on starlink, an IP address that starlink says is, well, we've geo located this to Yemen. Ooh, George Michaelson 34:09 So we might have arrived at a point where at entry, half an hour ago, we were in the world of Internet addresses don't actually do geographic address assignment because it just doesn't work for the purposes of Internet routing and distribution. starlink actually cared deeply about what is the physical location of the ground station in a cellular honeycomb map over the planet. Geoff Huston 34:35 No, no, no, you made the bad assumption they're not trying to tell you about the ground station, although that's a really interesting question. When you geolocate a ship at sea and the task is to assign it to a country code. What is the country code you should use? Ooh, is it the ground station? Well, no, because starlink do not only operate mirrors in the sky. They have they have laser links in space. So for example, to serve south or to serve the Sudan, the ground station is actually located, as far as we can tell, in Mombasa, in Kenya. That's what traceroute says. They beam it up [George: right] across by the inter laser links, and then come down to an area on the surface of the planet that starlink record as being in the country of Sudan. George Michaelson 35:27 Yeah, that's that's kind of the join over economic necessity and strategic requirements, because satellite technology is not something that just drops everywhere. Geoff Huston 35:36 I pick Sudan deliberately, and I could have used Myanmar, because if you want to double check what I'm saying, you kind of go, Okay, let's open up the starlink web page. It's so informative, not but they do have a global coverage map, and they clearly say there is no starlink service in Sudan and no starlink service in Myanmar, [George: right] But we're seeing 1000s of ads a day that starlinks own geolocation files map into the country of Sudan and the country of Myanmar. What's going on? [George: Right] Well, there's a clue from the Cook Islands, who recently signed an agreement with starlink recently, and it was pointed out in I think it's the Cook Island times. Isn't Google Search wonderful that users of starlink in the Cook Islands were being gently encouraged to cease their global. Unlimited roaming package that they purchased in the country of New Zealand. Yeah, and buy a service that was intended for the Cook Islands. George Michaelson 36:43 There's a quite intimate relationship between New Zealand and the Cook Islands in terms of provision of social service, health care, economic engagement. Geoff Huston 36:53 I don't think it's anything like that. I think it was just convenience. You could order a service in New Zealand and have the equipment shipped, maybe they'll folk to do it for you, for the simple application of money. And you could turn it on in the Cook Islands and presto, instant Internet. But you'd ordered a New Zealand service George Michaelson 37:10 Because of the underlying economic relationships. It was feasible to do that that actually happens across the border between Canada and America. AT & T and Verizon both have to occasionally tell customers we really don't want you to be buying that service. Geoff Huston 37:26 Mobiles tend to operate the boundaries pretty tightly. If you're in Switzerland and you carry the follow the boundary to France. You'll notice you switch mobile providers within a few 100 meters, but in this case, for starlink, no. So why does Sudan have so many so called users? And even Yemen? Ah, now the nearest countries to Yemen are Oman and Saudi Arabia. in Oman, the amount of starlink usage is point 8% George Michaelson 37:55 that's suspiciously low. Geoff Huston 37:57 No, it's not suspiciously low. I think it's accurate. It's not a cheap service. It's a good service, yes, but quite frankly, if there's a fine terrestrial provider, why bother with starlink? It's limited in a whole bunch of things, including spectrum. So starlink would be a second choice for most users, and that's reasonable. So why is it so big in Yemen? Well, there's no starlink in Saudi. none. So what do you think they do? [George: Zero]. I think there's some fine bunch of people who are selling them a styling service. And those fine bunch of people basically buy a service in Yemen and ship the dishy etc, over to the user in Saudi who sets it up, and the geographic attribution of that user, even though they're physically located elsewhere, comes up as Yemen. And then you look at that same coverage map, how much starlink is in, or at least how much official starlink is in Morocco, Tunisia, Libya and Egypt. No, they haven't gone through the regulatory approvals. So what do you think happens? Well, same thing. So it's kind of leakage, [George: yeah], so we've found sort of two problems here in Starling that seem to be contributing to this. George Michaelson 39:18 This is actually affecting your ability to use ad based data to capture a sustained history of growth and change, because your numbers for these economies are now projecting differences from the values you really would expect to see Geoff Huston 39:37 well. So what I did was compile a table of countries where the advertisement placement indicated that starlink had more than 10% market share, which seems to be, you know, pushing it uphill. [George: Yeah], Tuvalu 92% market share, Kiribati, 81% Yemen, the 59% there are 20 such economies. There's actually 21 but I'm going to make an exception of Pitcairn island. There are 54 people. Maybe one of them has died. There might only be 53 and yes, ads go to people on Pitcairn, God knows how. And yes, it all seems to be through starlink. I can understand that there's no cable there. George Michaelson 40:16 I would think Tuvalu is very plausibly actually using starlink. Geoff Huston 40:20 No, no. Tuvalu is an interesting case in point a there's a huge amount of ships at sea in the Pacific. It's a big ocean, and starlink services it in its entirety. There's always a satellite overhead, and with inter satellite links, you can bring the signal down almost anywhere that starlink chooses, right? And you go, Well, maybe it's ships at sea and not the domestic system? Well, yes, because recently, starlink announced a community gateway service. starlink always used to sell retail. You're a user, you make a contract with starlink. There's no intermediaries, there's no reseller. But in Nauru, they license. Senpac. And if you look at it, you will see a bunch of IP addresses where the path for routing is. It originates inside Senpac in Nauru and then uses a link, using starlinks network, number, AS, you know, 14593, whatever, and then re emerges in the Internet. George Michaelson 41:21 So you can actually tell when it's being used as infrastructure by a local telco provider, Geoff Huston 41:28 Right? In Tuvalu, the Tuvalu tel Communications Corporation is a community gateway. So domestic use in Tuvalu is presumably going to come from a different AS and starlink, [George: right] So, why is there 92% of people who are geographically mapped to starlink, you know, being located into Tuvalu? And my answer is, there are a lot of boats out there, and everyone George Michaelson 41:53 cruise ship, Geoff Huston 41:54 well, everything from cruise ships to individual yachts, and maybe a loony or two in his canoe. You know, who knows? And they've all got starlink. And what you're seeing is a distortion of these numbers by ships at sea. Is it just ships? Well, no, because what do you do about WiFi, Internet access in airplanes? And you kind of go, Well, okay, I just use the standard in flight WiFi. Oh, that's so slow. I'm using starlink. I can get a few 100 megabits per second down to the plane. Woo hoo. And so starlink is busy signing up airplanes. [George: Oh, yeah]. So we've got planes and ships distorting this data and mapping. George Michaelson 42:36 This is millions of users. This is not just six people on the boat. Geoff Huston 42:40 It's not noise anymore. It's enough that if you map to a tiny country like Tuvalu, it overwhelms the domestic data. So I think there are about 20 countries whose domestic numbers are overwhelmed by starlink. But this leads to, I think, some final, really interesting questions about geolocation. If I am a ship at sea or an airplane on an international flight whose law pertains to that craft, and George Michaelson 43:08 originating economy. Geoff Huston 43:10 No, it's the economy of flag. So if you're on a ship under a Liberian flag, and you're at sea on an international voyage, it's the laws of the flag state, not where you came from or where you're going to because it's an international trip. Now you can argue a little bit about territorial waters, 12 nautical mile limits and so on, and that's a fine argument, but in an airplane, when you take off on an international trip, the laws of the flag state of the plane apply once you leave the ground, George Michaelson 43:41 you choose your flag state for tax minimization, and Geoff Huston 43:45 I don't know what they do. George Michaelson 43:47 It might be badged American Airlines on the body, but the plane itself could be owned by Caribbean island state. Geoff Huston 43:55 Well, normally it's the operator, not the owner of the plane, but you know. But the issue is, if you're thinking about geolocation as political, IPR, rights, your hotel chain, etc, then this is actually a salient thought. Is it where you are or the National you know, legislative framework that applies to you? Is the question you're trying to ask Boeing when it did its service some years back, also had the issue that they thought at the time that the location of the earth station was relevant. So if, if the signal is sent from Kenya, then irrespective of where the aircraft is, is it a Kenyan system? Because on the earth, the packets went to Kenya and then disappeared upwards. Or is it who's receiving? Because the sender and receiver, particularly with starlink, with its laser links, are not the same earth station, necessarily. Yeah. Should all these be US addresses? Because as far as I know, SpaceX is a private company in the United States. That's its flag state. [George: Oh, wow], all this says, gee whiz. Now that it's no longer trivial, and it's not, and we're trying this sort of to map the old world. You live on the surface of the Earth, we assign you into a country, into a world that's not quite the same, [George: yeah], where we've got international waters, ships at sea, aircraft in flight. And these are non trivial numbers. What do you want out of geolocation and whom and how should it be managed? George Michaelson 45:30 Whose benefit? Geoff Huston 45:31 It, oh, my God, my brain has just exploded, because this is now not simple. And you kind of go, oh, it's only one or two people. No, I think it's actually, George Michaelson 45:40 no, no, no, no, Geoff Huston 45:41 a whole lot more than one or two people. You know, each aircraft has a few 100 people. And there are, you know, a lot of aircraft in the air, a lot of ships at sea. George Michaelson 45:49 And these are very, very well paid lawyers who are looking at the legal implications for intellectual property rights, copyright licensing fees, you name it. These are not people who are motivated to say nothing to see here. Geoff Huston 46:04 And so there is something to see, but it's not clear what the answers are. And so at APNIC, we've actually made a decision. When I say we, I mean I George Michaelson 46:16 royal we, Geoff Huston 46:17 the royal we. And I've come to the conclusion that my aim in presenting these numbers about the make up of the domestic market, in terms of which ISPs are big and small, I'm going to take those 20 countries where the starlink ad presentation rate is more than 10% and assign starlink to the mythical country "unclassified", because it's clearly not inside the country like in Yemen it's not the yemenis. It's ships in the Red Sea, and they're not Yemen or Sudan or even Egypt or anyone else. They're on the Red Sea, and that's not a country, [George: yeah], and so I'm going to take that out. Why? Because my purpose was to look at domestic economies and the market makeup, [George: yeah], so I feel entirely justified to move this source of distortion and simply kick it out of left field. George Michaelson 47:09 And the important point is, you're moving it. You're not deleting it, if you want to know, Geoff Huston 4&:14 oh no, no, no, I'm not moving it to the United States or anything else. I'm just simply saying I don't know where these things are, yeah, and the hints that starlink were providing me aren't reliable. George Michaelson 47:26 Are we going to see like a notch in the chart appear suddenly Geoff, or is this going to take a while to filter through? Geoff Huston 47:33 I use a 60 day sliding window. Over the next two months, the numbers will slowly normalize to a different number. George Michaelson 47:40 It's not a bad approach. I think that's really quite a nice way of dealing with this problem. Geoff Huston 47:44 Well, it's the best I can do, but it's a solution for me. What's Netflix's solution? I don't know what's Paramount solution. I don't know the Hilton hotel chain, and all of us, for various reasons, use geolocation, and we tend to use the maxmind database, the IPinfo.io, database, but slap a different interpretation. Some of them even use the RIR registration data, and it's kind of gets to a point of George Michaelson 48:13 or RFC 8805, the self published format that was proposed by Google is being used Geoff Huston 48:19 the lies that I tell other people. Yes, you could use that data, but it's just lies you don't understand. It's it's accuracy, it's just how I choose to present myself. It's not where I know it necessarily be. And so geolocation has all these fascinating facets to it, and what seemed like a really simple thing, [George: oh no]. Put this user into one of these 225 countries is kind of Oh, but I can't [George: Yeah], for a bunch now of statistically visible users, that doesn't work anymore. What are we going to do? Well, like I said, APNIC labs, we've solved our problem, and it's pragmatic. It doesn't feel the best solution, but it's certainly a solution for others. Well, you know, George Michaelson 49:03 too early to say, Geoff Huston 49:04 go, go hire some more lawyers, think about the problem and try and figure out how you're going to approach it. There's no instant answers here. George Michaelson 49:11 That's really fascinating. Geoff, Geoff Huston 49:12 that's a little bit of insight in into our measurement world and the kinds of things that come up, and how they how they appear. And I must, I must, with gratitude. Thank is a new member of the AFRINIC board who first raised this with me, Ben Roberts, I think his name is, is that right? Yeah. Ben Roberts, who said, Geoff, what's up with Yemen? I look at it go, Yemen is wrong. Ben, George Michaelson 49:29 well, reassuring the eyes are on the data and useful that you've come up with a way of adjusting your processes to take account of it. Geoff, I think that's fascinating Geoff Huston 49:39 if anyone sees weird stuff in the APNIC labs data. You know. Don't keep it to yourself. At least tell me and we'll investigate as to whether it's you know, us or the world that is at fault. George Michaelson 49:49 That's great. Thank you. Geoff Huston 49:51 Thanks, George. Thanks. Listeners. See you. George Michaelson 49:55 If you've got a story or research to share here on ping, why not get in contact by. Email to ping@apnic.net or via the APNIC social media channels. Also remember the measurement@apnic.net mail enlist on orbit is there to discuss and share relevant collaborative opportunities, grants and funding opportunities, jobs and graduate placings, or to seek feedback from the community on your own measurement projects. Be sure to check out the APNIC website for all your resource and community needs until next time you