Geoff Huston 0:00 Well, we kind of wanted to do the same thing in networks, because wouldn't it be good if we could take the service, the content, and pre-distribute it closer to consumers who wanted it, and then when we say it's here, it's everywhere, and do we need to even wait and pre-provision? Can we use the secret of things like the DNS itself, where the first answer gets copied by the DNS infrastructure in computers close to who's asking, close to but not the same, so all your other customers in that network who ask for the same domain name shortly after the first simply get a copy of what the first answer was. What's the IP address of google.com Oh, that's boring. I've already done that 20 times today. Have a cache copy. George Michaelson 1:01 You're listening to Ping, a podcast by APNIC, discussing all things related to measuring the Internet. I'm your host, George Michaelson. This time I'm talking again to Geoff Huston from APNIC Labs, in his regular monthly spot on Ping. In several recent episodes, Geoff has asserted the routing function in the Internet has moved from IP level packet forwarding models into a process driven in the name to address lookup function. Who you are and where you might want to go is no longer just about your IP address and what you might think is the other end across a network, it's now about where you are and where intermediary services can best service your request from, but how does this really work in practice? What kinds of processes determine where you should be served from, and who decides? IP level routing hasn't gone away. There's still a very important routing method called Anycast for selecting the closest server to be used, and this is deployed at scale, but it's increasingly being augmented by name-based DNS- based methods to determine where to go. Geoff, welcome back to Ping. Geoff Huston 2:17 Hi, George, how are you today? George Michaelson 2:19 I'm good, Geoff, but I'm confused because you have spent so many of your recent ping recordings talking about how names are replacing routing as the steering artifact for the global Internet, and I keep meaning to ask, how does this work, Geoff Huston 2:36 right? So let's get into that, because I actually think this topic is totally fascinating, and I actually think it's revolutionary in the change that it's making to the concept of computer networks, you know. Computer networks were, I suppose, a lot like telephone networks. I'm here, you're there, and the job of the telephone network is to actually carry my voice to you and carry your voice to me. [George: Yeah], I can't teleport you, I can't change you, I can't duplicate you. The network is there to bridge the two of us together without either of us getting out of our chairs. Great. Now, when we built computer networks, the first thing we did is build it in the image of the thing we knew. Didn't have any other model? [George: Yeah], so we kind of built them as a way of having conversations between computers. It's over there, mine is over here. You know, I tap on my keyboard, little packets get scurrying over to your computer. You tap on your packet, your keyboard, little packets go scurrying back. George Michaelson 3:39 It's kind of inherently, me to you, you to me. It's not typically a form of communication that's inviting the idea of me to 16 other people or me to two people at once. It's building it point to point, you to me, me to you, Geoff Huston 3:55 right. And, oddly enough, you know, there were these other models, and we have their paradigms to some extent, you know, radio and television is a form of broadcast where everyone within a certain radius can listen to a single point source, and the source is available to anyone within that served geography, and multicast is slightly more selective, it says I wish to simultaneously get a bunch of people to listen, and I'd actually like to agree that they're listeners, but you know, we have computer analogies, computer networking analogies of those other models as well. Although fascinatingly, they were really hard, and they never took off in the mass market. [George: Yeah] Geoff Huston 4:40 you know, part of the problem with multicast is that it was, to be perfectly frank, a really fragile technology, and the only folk who tried to use it were actually folk who wanted to have a digital version of radio broadcast, and the idea that the audience would all line up at the same time to listen. Listen to you seemed a bit far-fetched in today's world. They don't do it like that, and so it never really caught on. George Michaelson 5:06 Well, I've just watched the launch of Artemis Two on the start of its journey, mankind reclaiming outer space one rocket at a time, and because of the nature of an event that is happening at a set time, there are still things - sporting functions, opening of parliament, royal marriages - where people want to line up at the gate, but if you're talking about the vast majority of digital content that we used to see as a collective engagement, it's no longer available at 9pm and only at 9pm it's you watch it when you want it, so that model of let's get everyone together in time and say "go", it kind of narrowed down, Geoff Huston 5:51 it's gone, George Michaelson 5:52 it's gone. Geoff Huston 5:52 I didn't see the launch, I'll catch it up later on streaming, George Michaelson 5:56 ha, Geoff Huston 5:57 me and billions of others, no doubt, so yes, that paradigm of simultaneous audience, I think, doesn't exist anymore. I think the problem was, in so far as the original version of computer networking was actually small transactions, they weren't large ones. It was more about replacing the office fax with electronic mail. It was web pages that were realistically short, sweet, and simple. George Michaelson 6:26 Yeah, Geoff Huston 6:26 images rather than movies. George Michaelson 6:28 If we can make the event of getting it really quick, you can get on with reading it at the speed of human eyeballs, and me, the network, I don't have to sit there, I can deal with someone else's problems, Geoff Huston 6:40 right? So the task of the network was simply to carry the consumer, the user, the client to the server, where whatever it is they wanted was, and carry the answers back. And by and large, this served us for a couple of decades, but as our computing technology got better, our displays got better, and we were digitizing almost everything. What actually happened, of course, was that computing was getting more interesting. You could do bigger things. George Michaelson 7:13 When I transited from Britain to Australia, this was around the time that Ken Thompson, one of the two people who built the Unix operating system we all live and depend on nowadays. He was one of the privileged people who had a server at home with a 600 megabyte disk drive that he had dedicated to recordings of music, because he was exploring psychoacoustic effects under compression of audio, and I remember having conversations with people saying you couldn't afford to send music like that down a network, if you had a modem, it would take a month to get even an hour's worth of music out of it, but the thing is, he was researching something that, as you've just said, has come into the fore as computers have got faster and as networks have got faster, we can now do things that we simply couldn't imagine when this form of networking was being invented. Geoff Huston 8:07 So, yes, computers have got faster and cheaper, storage has got cheaper, 600 megabytes. I'm sorry, a significant problem today is 600 petabytes, a major problem is 60 terabytes, but under that, get into the store and buy a bit more, either spinning rust or or your solid state storage, and you're done. So, in some ways, we're living in a world of abundance, but one thing hasn't changed, and oddly enough, that's the speed of light over distance, and when you think about computer protocol performance, distance is a pain, because the way we actually make networks work is a feedback loop. Oh, go a bit faster, there's room in the network, says your feedback, and it's this delay between desire and reality. If you stretch it out, it's like living in a world where everything happens a second later, you have to slow yourself down, because you do something, you have to wait for a response. And the issue was that, although we had heaps of bandwidth, heaps of computing, heaps of storage, we still had this problem that distance was killing us, George Michaelson 9:19 right, because the effect of the speed of light means that the feedback loop you to me to either tell me speed up or slow down, that message necessarily has the delay of the speed of light between you and me. The further apart we are, the more delay there is between you saying stop and me actually stopping, which means my responsiveness to changes in the network, whether it's full or empty, congested or noisy, is necessarily bound in that delay. Geoff Huston 9:50 So it's really hard, oddly enough, even though we have high-speed networks, very high-speed networks, it's really hard for a user, a client. To actually exercise that, so if I want to stream a video from I don't know, I live in Australia, so stream a video from the United Kingdom, from France or Germany, from the other side of the world, even though every individual component might be megabits of capacity and there might be available megabits of capacity, I wouldn't get it, and this was kind of the issue that was confronting originally folk like Microsoft and Apple. Their problem was that they released operating system updates every couple of months and said, "Right, there's a new update for Windows available, come to Seattle with your network and pull it down from our servers. Now, nothing wrong with that, other than a few 100 million people wanting to do so simultaneously. George Michaelson 10:49 Yeah. Oh my gosh, the Windows releases midnight in America. I'm going to wait till one minute past midnight and fetch it. Geoff Huston 10:57 You and 100 million of your best mates, and so things melted and didn't take us long to figure out. Well, if computing and storage is so cheap, let's replicate the content in advance. It's like in the old days, a book publisher would send in advance copies of Harry Potter and whatever it was to bookstores, and then announce the opening day, and the stores had already been pre-provisioned with the book. George Michaelson 11:23 Yes, there were a few lawsuits where the stores were selling pre- release copies of the book under the counter to journalists, and people were breaching the magic wall of Geoff Huston 11:31 The Secret Covenant, or whatever it was. Well, we kind of wanted to do the same thing in networks, because wouldn't it be good if we could take the service, the content, and pre-distribute it closer to consumers who wanted it, and then when we say it's here, it's everywhere, and do we need to even wait and pre-provision? Can we use the secret of things like the DNS itself, where the first answer gets copied by the DNS infrastructure in computers close to who's asking close to, but not the same, so all your other customers in that network who ask for the same domain name shortly after the first simply get a copy of what the first answer was, what's the IP address of google.com Oh, that's boring. I've already done that 20 times today. Have a cache copy. George Michaelson 12:28 Cache, the magic word emerges. Caching. Geoff Huston 12:32 Well, I'm actually thinking it's distribution. Caching is what we call it in the DNS, but in the server world we call it content distribution. That's what you're trying to do is to take a big thing, be it a video or any other kind of large volume server or server's content, and as you stream it outwards, you keep a copy close to, but not right inside the end user's environment in their ISP, for example, so now when anyone else asks for the same video, and I bet you they will, I'll be able to serve it remotely and not touch the network. Brilliant, all of a sudden I've got rid of the network, it's gone. George Michaelson 13:14 Well, you needed a pipe, you could push the stuff down to get it to all those places you're choosing to put it, so "a" network has to exist for you to push stuff to the places you want to cache copies of it, but "a" network isn't the same as "the" network, is it? Geoff Huston 13:36 Well, let me make your brain explode, then, because let's Geoff's content distribution system rig up a bunch of virtual servers on borrowed hardware or leased hardware in a few 1000 places all around the world, close to populations, and they will serve Geoff's content locally over the Internet protocol. Fine, how do I feed those front ends? Well, I have the Geoff content factory somewhere else, doesn't really matter where, and I want to feed it in an advance. Oh, that means I could trickle feed it. I don't need to be fast, it's not on demand, I'm pre-provisioning. What protocol should I use to do that pre-provisioning? Apple Talk doesn't matter, George Michaelson 14:28 private, Geoff Huston 14:28 you don't see it, George Michaelson 14:29 it's private, Geoff Huston 14:30 it's all private, it's private, and whatever protocol I choose to use is kind of my business between my master server and my front- end, you know, retail points of presence, my content points of presence. So, in some ways, all I'm trying to do is deliver this same content all over the world through the last hop, but how I do so internally is kind of my problem, and no one else's, and so kind of doesn't matter, and so. Google internally uses BBR or IBM SNA. No one cares. It's not our problem, it's their problem. George Michaelson 15:09 Yeah, from talking with the engineers who manage the FreeBSD distribution framework, I think they are existing in the margins where they have to try and reduce cost here, and they've constructed almost, I believe, a two-tier model. They have regional nodes that they push to, which means the center only has to arrange for maybe five copies to be pushed out, and then local nodes do a pre-fetching, a pre-provisioning fetch from a regional node. So they've kind of invented two levels of behavior, and they've got less complexity, but in the end, nobody in the real world has to care, because all they see is fetch-free BSD from a local mirror. Geoff Huston 15:52 You could be talking about Akamai, or Fastly, or Google, or any of the others. They all use the same techniques. It's really common, and it works. That's a good thing. It works. So, okay, distribution is kind of a solved problem, but let's go to the other side of this. You want to go to Geoff's favorite videos from Geoff's Content Distribution Network, and I have 1000 points of presence now. If you make a dud choice and pick a server that is not near you, but on the other side of the world. No one's better off, are they? [George: No] you're going to have a bad experience, because that was a bad choice. George Michaelson 16:32 I feel like this has been the longest preamble in the history of podcasting, Geoff, because we've just arrived at the entry point. How, how does it work? Geoff Huston 16:43 Well, I was wanting to kind of motivate it, and George Michaelson 16:45 you got there Geoff Huston 16:46 everyone to understand the nature of the problem and kind of why it's not straightforward, and there are a number of techniques, if you will, to actually answer this, and the first approach, which oddly enough was very controversial, was actually to put the same computer platform everywhere that you're serving, so this computer platform uses address 192.168 dot 0.1 and I put another one in Paris, 192.168 dot 0.1 and I put another one in New York, one in Sydney, one in Singapore, one in Beijing. I use the same address everywhere, and I go to the routing system and go, Hi, I'm over here, and here, and here, and here. George Michaelson 17:33 Now you've been at some pains when we talk BGP and the behavior of BGP selection of a prefix to route to, you've talked about how, if you announce a big address and then announce some more specific wins, that becomes the thing you look at, and separately, you've talked about the loop detection mechanism becoming a cost measure that lets you determine which is the lowest cost path to take, and I'm looking at this situation where you're announcing exactly the same address in BGP in lots of places, and I'm asking myself, what mechanism makes me pick one near me and you pick one near you, and some Parisian pick one in Paris. Geoff Huston 18:18 Well, you said it yourself, BGP picks the route with the smallest metric. Now, in BGP, the metric is the number of networks I need to traverse to get to where it's being announced. So, if this conversation is you and I sitting in, say, America, and the server is being announced in, say, Paris. It is likely there might be a local network, a transatlantic transit network, and another local network. It's likely the cost to get to the Parisian server is three, [George: right] But I've also got my server in America. It may well be that it's very, very close to you, an adjacent network cost one George Michaelson 19:05 right, Geoff Huston 19:06 BGP picks the lowest cost, and so if you go to 192.168 dot 0.1 and you're in America, you will go to the server that the routing system BGP thinks is closest to you, George Michaelson 19:20 so in a sense it's almost like a decision-free outcome. I don't have to pick where to go. Rooting optimizes the decision facing me and the guy next door, and simultaneously some Parisian guy sitting in a flat. We all get optimized in rooting to the closest point, Geoff Huston 19:40 right! And you go, geez, that sounds weird. Let me point out a few major uses of this anycast approach, where you put the same address everywhere, and I point to the root of the DNS. The root of the DNS is served by 13 unique V4 addresses. And 13 unique V6 addresses, George Michaelson 20:02 right. Geoff Huston 20:03 Each notional server has a V address and a V6 address. So, there are 13 root servers. There are 13 machines that serve the root. No, there are 1500 of them. Well, how does that work? Each root server actually has a large number of servers, servers all over the world, all over the Internet, all listening on the same IP address, the same, so there are 13 separate little what we call anycast clouds in v4 another 13 in V6 and the routing system automatically takes you to the closest server. No one does anything. BGP looks after all of this, and it's actually not a bad approach. You've kind of outsourced your problem to BGP, knock yourself out, kiddies, you know. So not only does it keep the root of the DNS evenly loaded across the world, and stops the root servers from melting through overuse or any individual server. The technique also works for efficiency. You just get taken to the closest one. Who uses that technique? Cloudflare. So, Cloudflare have a very small number of IP addresses, but if you look hard at their network, you actually find the same IP address pops up everywhere, because they're anycasting and therefore direct you to the closest content on Cloudflare's network is certainly left to BGP, easy. George Michaelson 21:36 So nothing comes for free, Geoff, and I'm going to imagine there are potential pitfalls in this, I mean, I'm sitting here thinking, if one of these boxes breaks and I need to log into it, I can't use this magic anycast address to log in. There's got to be at least another address on it that is unique to each, each box. Geoff Huston 21:53 Oh, you need a, you need a service address that's unique, but let me tell you what the problem is. The network, in topology sense, is not long and stringy, is short and fat. The average AS length on the Internet is a little over four networks, four, so the average AS metric is four, which is really coarse, and if you attach to a big spanning network, so if Geoff's ISP spans all of America and Europe with one AS [George: ah right] Then, in terms of BGP metrics, every location in Europe and America is the same distance. How do you know what's closest, George Michaelson 22:36 right? Geoff Huston 22:36 And the answer is BGP doesn't, and so that will give you poor outcomes if you are dealing with networks that span large distances, and guess what, there are networks that span large distances, and at that way anycast kind of makes, unfortunately, suboptimal decisions, and we see this. George Michaelson 22:59 right, you could wind up being in America with a resource in France looking equal cost to a resource quite close to you logistically on the continent, and you could incur that time delay of a sub oceanic fiber optic link, because BGP can't tell you you should look somewhere else, Geoff Huston 23:18 right? And the right research folk, one of their early experiments with Atlas and mapping was to actually ask their Atlas nodes, which instance of A.root-servers.net do you go to when you ask the root of the DNS, which instance of B, and so they were trying to actually isolate how well does this root server any cast system actually work, and obviously when you go looking for anomalies, you find the world is full of them, and of course, there were Atlas nodes in Europe who thought via BGP that the closest anycast instance of one of these servers was in places over the other side of the Atlantic in America, and so on, and it wasn't rare. It's common, because BGP is pretty coarse, you know. Short fat networks don't really do a good job with anycast. Okay, George Michaelson 24:12 and if I front up on a big network and say, hey, could you make your network a bit more granular? Could you grind it up a bit to divide it into the chunks that would make my life easier. The inevitable point is, yeah, I could do that, Geoff Huston 24:25 but you've got to pay me. You've got to pay me large amounts of money, and the real answer is, I don't think you're going to, because I'm not hosting your content, and I'm not hosting your customers, I'm just the transit dude in the middle, and so the economics don't really make this any better, and trying to make a longer, thinner network out of a short, fat network is kind of anti-gravity. No one wants it, so it's not going to get any better. Oddly enough, it's going to get worse as we move on. So, in some ways, anycast has its limits, and if you really want to start shaving things down. If you really think there should be a difference between Frankfurt and Paris, if there should be a difference between Paris and Lyon, there should be a difference between Paris and what's a town very, very close to Paris, Paris, Orleans, a few kilometers away. If you really want to make that difference, or a distinction between Tokyo and Asaka BGP isn't going to do it, it's not George Michaelson 25:23 right. So, BGP was almost the decision-free mechanism where you, the content provider, and me, the client, have nothing we had to do to find what was considered the air quotes best choice, and what we're kind of getting to is, well, you actually are going to have to make a decision now, and there's that magic moment. Who's making the decision? How do they do it? Geoff Huston 25:46 So, if I can't use anycast and get a free ride with routing, let's try a different approach. So, again, I have my content distribution network, and I give each of these points of presence a different IP address, 192 168 1,2,3,4,5 so let's have 1000 of them. You asked for Geoff's content.com Now I've got to go. Oh, I could give you in the DNS 1000 different answers. Which one is the answer that's good for you? And you put your little hand up, going, well, I didn't really want all 1000 answers, I can't cope with that, not my problem, but don't give me the one that's furthest away, please, just don't do that, Anchorage, Alaska is only good if you're living in Anchorage, Alaska, George Michaelson 26:36 So I trigger a decision having to be made because I say I want to get somewhere, and you, because I'm getting somewhere by asking about, tell me the address of a name, your systems, or someone on your behalf is getting that call that's saying, I, George, I want to do this, and the problem is, How can I know based on you asking where the best place for you is? What do I know to help decide. Geoff Huston 27:03 So, let's make a sweeping assumption, and it's probably not that bad a sweeping assumption. You and your DNS agent, your recursive resolver, the thing that actually asks your questions for you is normally provided by your ISP. Let's assume that that's close to you, so now I have a third party, your resolver, and I have it asking me, the authoritative server for my content distribution network, a question about content, content that I'm serving, so now I've got a simple problem, or simpler, an IP address is asking me about a server. I have 1000 possible IP address answers for that name. I just need to pick the one that's closest. Okay, we have geolocation databases in its crudest sense. I can map to countries. Let's put a country code on your resolver that asks the question. Fair enough. You live in Japan. Let's shuffle through my list, find an answer where the server is located in Japan. That's the answer. So now you're getting steered to something that I think is good. George Michaelson 28:13 It definitely sounds like an improvement against large flat networks that potentially span interoceanic networks and different economies, but I'm not feeling like you've kind of got perfection yet, Geoff, because some economies are very long and thin. Chile is a massive, long, thin stripe down the side of South America, and if you happen to be right down the bottom at Ushuaia, you are not close to Santiago, right up in the north, Japan is a very long, thin island. It's quite a lot bigger than people think, because of the Mercator projection. And if you're right up the north, you are not close to right down the south. So you fix the problem by keeping it within Japan. I'm giving you a thumbs up. It still hasn't got close to me. Geoff Huston 29:00 Well, before you start throwing rocks, let me observe that this is the model used by Akamai, one of the major content players on the planet. They do this by DNS. It's not even a clever trick, it's an obvious thing. When you serve content through Akamai, when you say take this content and serve Akamai, they say to you, look, instead of serving or giving an IP address of your server in the DNS CName, it alias it to us, and so they say, if I want to serve example.com they might say, in your DNS master zone, use the CName for this service, example.www.example.com.www.example.com and C name it to dub dub dub.example.com dot edgekey.net Who owns edgekey.net Akamai. Oh, now I've got dub dub dub.example.com dot edgekey.net Where does that. That map to another CName. Now the trick is that that second mapping depends on who's asking, because I have maybe have 4000 possible answers that identify an edge location by name, George Michaelson 30:17 right? Geoff Huston 30:17 And I do a magic thing that says www.example.com dot edgekey.net CNames to the server in Tokyo dot akamai edge.net What's its IP address? Well, that's constant, that's easy. The server in Tokyo dot akamyedge.net you know, is 23.1 95 dot 84.2 41 or whatever, George Michaelson 30:39 right? Geoff Huston 30:39 And so now with just a little bit of mucking around with Cnames, I've got a decent way of doing this, as long as I can triangulate and say, "A, you live close to your DNS resolver, so if I optimize my answer against who's asking, I can steer you to one of the servers that serve the content you're after that's closest to you, George Michaelson 31:04 so having kind of kibitz and said yeah, don't like this, this don't think it's perfect, I want to back off and observe, I'm sure I'm using Akamai services all the time, proof by example, I'm not seeing massive amounts of "this kind of sucks", so as a consumer of services, I think we might be in 80-20 land. Geoff, this works mostly if I'm not doing stupid tricks with my DNS, and if I'm mostly using a DNS server close logistically to where I live and work. It basically seems to work. Geoff Huston 31:37 Many folk might bank with the Bank of Akamai, they might, it works, and the beauty of this is also that you can't see those translation names when your browser then connects to this IP address of an Akamai server close to you. You say, Hi server, I want to connect to www.my bank, and I want to create a TLS connection, and the Akamai server goes got a problem. I am configured to be an alias for that banking name. Here is a key, here is a key exchange, here is a TLS connection. Let's do this, so the entire thing is seamless to the user, and I've now managed to by doing a simple sort of trick with the DNS, steer you to something is really close to you. Presto, it's fast, it's cheap. Did I mention it's fast? It's really fast. George Michaelson 32:31 Yeah, Geoff Huston 32:31 because it's local, so kind of works. Yes. So now, George, let's start throwing rocks. George Michaelson 32:38 Yeah, so I want to watch content from the BBC, and the BBC says, 'Sorry, chum, your DNS is in Australia. So I ring up a mate with the DNS server in the UK, and I let him let me ask questions through him, and the BBC go, 'Not a problem, buddy. I see your questions coming from the UK. Here is Doctor Who'. It feels like I can drive through geo-fenced IPR rules if I can make my questions come from the UK Geoff Huston 33:10 more than 10 years ago with Netflix. If you changed by simply going into the config part of your computer and changed who your DNS resolver was to your country of choice, Netflix would open up the library of that country because it assumed you were your DNS. George Michaelson 33:27 Yeah, Geoff Huston 33:28 and yes, it's it is, was, and probably still is abusable from that respect. George Michaelson 33:34 Yeah, Geoff Huston 33:34 and the other problem that's just as bad are these open DNS resolvers, George Michaelson 33:40 yeah, Geoff Huston 33:40 like 1.1.1.1 and 8.8.8.8 Well, the thing is, which actual DNS engine did your query go to? You don't know, but when Akamai get the query coming from one of these open DNS resolvers, say Google, it's not clear that the address being used by that open DNS resolver, when it asks Akamai is close to you. Oh, you're coming from Paris. Oh, you're coming from Moscow. Oh, you're coming from Madrid. You know, it's really, really hard to nail it down. [George: Yeah], and so under some pressure, Akamai and a few other content folk convinced the big content distribution networks to add a security leak, a nightmare. When you ask the authoritative server, why don't you attach the client address? Oh, can't do that. Don't want to do that. Why don't you attach the client's subnet? Oh, that's okay. That's not very privacy leaking rubbish. We'll use that, so we actually intrude into the DNS between the recursive and the authoritative an item of data that actually points to the original user, George Michaelson 34:51 right? So the person asking the question is the primary vehicle that can at least target in a lot of cases down to an economy. Economy, but we've changed the protocol to intrude a flag going out asking the question, this geezer in this suburb in Ginza is asking about this TV show, and the belief I have is, oh, well, you just identified me to Ginza, there's a million neighbors, and the belief you have is nobody's lying here. He really wants the server in Tokyo, not the server in Hiroshima, because he said Ginza, so it kind of sounds better, Geoff. Geoff Huston 35:30 It kind of sounds better, because in essence, you're giving a few more clues of information. Okay, how good is a subnet? It's a privacy protection. Meh, not very good, you're leaking information about you that maybe you didn't want. The plus side of this, who controls the mapping from name to number? You see, in BGP Anycast, the service provider had absolutely no control, George Michaelson 35:57 right? Geoff Huston 35:57 BGP, but here I am, my DNS servers control that mapping, I can twiddle and play if the server in Asaka is flat on its back, howling at the moon, and the only available capacity is somewhere else, Tokyo, whatever, I can actually intrude into this, and with a short life answer, I can tell you somewhere else, George Michaelson 36:21 short life, that would be because these kinds of answers wind up having a cache behavior, and if I've cached in your system, you have to go over here for the lifetime of the timer, I'm going over there. Geoff Huston 36:34 The thing that makes the DNS work so well, caching kind of is your enemy if you're trying to do some degree of dynamic load balancing or optimization, because once you give an answer in the DNS to that resolver, everyone else who queries for that name is going to get the same cached answer, George Michaelson 36:53 right, Geoff Huston 36:53 and if you want to recompute, then the only thing is give it a short lifetime in the cache, because otherwise the first answer is sticky for an unknown number of subsequent people, and so to do this, you lose cache. George Michaelson 37:07 So, these are two different kinds of rocks being thrown here. One is it's a bit of an anti-pattern for efficiency in the system. Okay, we have to deal with that, and the other is it's a big privacy concern. You don't really get to keep your privacy when you ask these questions anymore. Geoff Huston 37:24 Yeah, tough. So, what do you do? Well, we've just discussed, I think, almost the transition of networking and technology. We originally started with a smart network and dumb devices, it was called telephony, and we started to put more and more smarts in the edge devices and strip out the middle, so using BGP kind of relies on magic, and BGP devices need to do very, very little. Using the DNS lifts it up one level, so now you've got the DNS doing this distance mapping and trying to get you to the optimal point. Why stop there? You could do what Netflix do. There is an Amazon server, or 20, or 1000 where all the initial transactions for Netflix content get passed to on the West Coast of America. Oops, but that's okay, because then there's an application problem, because you divide your content into manageable chunks, little bits, and you have the ability to go well, user, you're making contact with me, and you want this content. Now, it's not a DNS question, it's not even an IP routing question. Now it is a content streaming question in the application level, George Michaelson 38:43 and it's between me, the user who is paying you Netflix for this service. I'm giving you money, and so this anonymity and privacy thing. Well, I was never going to have any privacy in you, because I was always going to pay you to give me the things that I tell you I want to get, Geoff Huston 39:05 and you want me to know it's you, because you're paid me, and you want that service, so you know George Michaelson 39:10 I do. Geoff Huston 39:11 This is now consensual disclosure. George Michaelson 39:14 So, there's a third thing kind of buried underneath here, which isn't really material for the technique we're talking about, but I think it's part of the story. You also, when you send me things in Netflix, you send it fast enough that I can build up quite a big buffer in time. I get content from you that spans probably two or three minutes in me, so that even if you have to stutter a bit sending stuff to me, I don't see that watching this movie, and so that chunking technique, where you're saying, "Oh, that server's dying, I need to move you somewhere else". In your clients, you know, most of them have a bit of a buffer here, where if they have to flick open a link somewhere else and start dealing with it, it isn't actually going to affect their streaming. Geoff Huston 39:59 Everything is a compromise, isn't it? You can do, and for a long time we did do high watermark, low watermark, trying to get the playback buffer in the device that you know the customer is using, the television, if you will, trying to keep that buffer full, and when it got to a low mortar mark, you just slam data down the line, and then went quiescent until the playback got to the low watermark threshold, we call that the Netflix spike, and it's a bit like hitting the network with a sledgehammer once every few minutes, and it's just as painful. Modern thinking says don't stress the network like that, bad things happen, you just drive the network into bad places, try and even your load, [George: right] Try not to do stress techniques, but do the gentle feed. Yes, it makes it a little bit more difficult to change where your stuff is being served from when you need to, but at the same time the network is going to behave a whole lot better. The good news about this approach, because now it's at the application level, and I have control over who is delivering you elements of the service. Is I can start playing with what if questions. George Michaelson 41:10 Yeah, Geoff Huston 41:10 I've been serving you from Asaka for quite some time now. I wonder if Tokyo could deliver you better service. We're up to chunk 1023 Let's deliver 1024 from Tokyo, and tell me, because it's my software at the other end. Tell me, how that was for you. George Michaelson 41:27 Yeah, Geoff Huston 41:27 and I can then dither and try and get really fine-grained accuracy on what is the best service at this point in time, George Michaelson 41:38 and so this mechanistic behavior, it might be using intermediaries, it might be using locally best service anycast in all kinds of weird ways, but we have moved back into application layer decisions between you, the sender, and me, the recipient. It's an instrumented dialog, but you get to understand it without intermediaries having to make inferences. There's ways that you can control the behavior, and I can control the behavior in software. Geoff Huston 42:06 So, as we've moved content up into video, as the content stuff gets bigger, and we're now talking about, you know, hundreds of megabytes or gigabytes, then the whole idea of make a decision once on what's best and stick with it, which is Akamai's decision doesn't really work very well, and so you have to constantly use some kind of adaptive feedback to check if that original decision could be bettered if there's another way of doing it to improve the response, which is where Netflix and YouTube and a number of other streaming platforms have gone to, because it becomes an application issue that can best actually address this problem. George Michaelson 42:43 So, let me just run history on the quick track. We had its point to point, and you could even have operators plugging between patch boards to connect you to me. We had its packets, and we're going to give them addresses, and suddenly it's about knowing how to get there, which means routing. We had routing can be made to look like one thing is anywhere, and you have to use a routing trick to pick which is the best one. [Geoff: Yes], we then say that isn't good enough. Can we go back to something that's about names? Who am I, and what do I want to get? And we start using mechanisms in names, and you've now moved up to well, it is names, but it's also got a lot of what are you actually doing with me? Let's measure this, let's optimize this kind of sounds like it's name-based, it's a bit of the routing story still in it, but it's also smarter. Geoff Huston 43:37 Well, it's kind of pushing the problem further and further up into where there I think there is intelligence and information to make it so the DNS problem is I have control of that translation, I can select a server for you, but then we're both stuck in that selection, we can't change it easily. When I start to dither at the application level, I have a huge amount of flexibility, and I can actually figure out was my original decision good or bad, and if it's bad, are there alternatives that can offer a better service? Don't forget, too, that there's a lot of remedies. I can reduce the quality of the signal to reduce the volume, or preferably I can stay at the same playback quality, but move to a server that has a better chance of keeping that quality up for your experience, and so that flexibility sort of plays into hands of the video streamers going, I'm really interested in user experience and the quality of that, I think I could do a better job. What does this mean, I suppose, for interoperability and standards and all that kind of stuff. The answer is no, not really anymore. George Michaelson 44:45 No, this is moved out of the domain of let's do it all the same way into I've got special secret sauce. Geoff Huston 44:53 There's Geoff's secret sauce, there's George secret sauce, and a playback from Geoff's server might not necessarily. Necessarily work with George's receiver, it's not that kind of interoperability anymore. George Michaelson 45:06 Yeah, Geoff Huston 45:06 each of these application level content factories and distribution systems work in their own ecosystem. George Michaelson 45:14 Yeah, we're building vertical markets and vertically walled gardens. You want to do this behavior, it's going to be good in me, and my special sauce makes it better in me than from the other chaps. Stay with me, Geoff Huston 45:27 right? And it kind of goes well, but I'm a minnow in this world, I'm just a starter. Oh God, I'm sorry. You're back to banging the rocks together, dude. You know, you're doing a web-based playback buffer because you're not big enough to build your own ecosystem, [George: yeah] And it kind of does play into a centrality argument, yes. But on the other hand, it also says you can do a better job, but you, you need to be able to reach deep inside the applications and the systems that are making this happen and customizing it for your needs. [George: Yeah] and that's sort of where we stand in this, that you know, it's a game for a small number of high-tuned approaches to this problem, delivering experiences I think that are amazing, but limited to a few large players to actually optimize that. George Michaelson 46:16 Yeah, so we've kind of got where we needed to be, name-based service delivery efficient and local, but along the way we also bumped into some realities in the modern world. Nothing is free, and this behavior has emerged in a way that's tended to concentrate market power a bit. On the other hand, as consumers, we're making decisions all the time in this environment, Geoff. Maybe this is something that we have to be prepared to do, eyes wide open. Geoff Huston 46:46 As consumers, we make choices. Yes, and this is all driven by those choices. I want better quality video, I want quality that's better than what I can even get from local broadcast, and so on and so forth. And that's what drives this, and I suppose the observation these days is, if you look at the proportion of traffic volume, and even the proportion of consumer money, we are a video streaming network streaming down the last mile, that's what the money says, everything else is a marginal, a marginal behavior with a few percentage points of revenue on the side, George Michaelson 47:23 yeah, Geoff Huston 47:23 and of course the big volumes drive the industry as a whole. This is what we invest in, so very capable, very fast last mile networks, g 6g fiber, you know, you name it, that's what customers want, and a densely driven local data center network where the content is replicated densely at a level of cities, not ISPs, not countries, at a level of cities of population dense points. That's what we're building at the moment. And you go, well, how does this work out in the future? I don't know. AI data centers are so big and expensive, we're back to a couple per country, if you're lucky, you know, the power systems, and so on. George Michaelson 48:04 Yeah, Geoff Huston 48:05 I suppose the real answer is we really don't know what we're building. George Michaelson 48:08 No, Geoff Huston 48:08 but we're spending a lot of money building it anyway. George Michaelson 48:11 Yeah, Geoff Huston 48:11 and that, I think, is another topic. The AI, you know, is this a boom and about to go bust, or is there something behind it? But we can talk about that. George Michaelson 48:21 That's a story for another day, Geoff. Geoff Huston 48:23 Another day, George. George Michaelson 48:24 But this has been absolutely fascinating. Thanks, Geoff. Geoff Huston 48:28 Thanks, George. Thanks. George Michaelson 48:31 If you've got a story or research to share here on Ping, why not get in contact by email to ping@apnic.net or via the APNIC social media channels. Also, remember the measurement@apnic.net mailing list on Orbit is there to discuss and share relevant collaborative opportunities, grants and funding opportunities, jobs, and graduate placings, or to seek feedback from the community on your own measurement projects, be sure to check out the APNIC website for all your resource and community needs. Until next time.