Geoff Huston 0:00 The idea is that if you're going to take a punt on Moore's law, if you're going to pump whole lot of money into believing that a technology will become viable in three years time, that isn't out here now, and you're going to pump a whole lot of money into a service that you're trying to build on the fly because you don't quite know how it's going to work, then the last thing you want is to have someone nipping at your heels, and so you tend to fixate on a different model of venture capital funding. That model is "winner takes all" - competition is for losers. George Michaelson 0:45 You're listening to ping, a podcast by APNIC discussing all things related to measuring the Internet. I'm your host, George Michaelson, this time, I'm talking to Geoff Huston from APNIC labs again in his regular monthly spot on ping Geoff and I talked about centrality and decentralization. He's just back from a very interesting session held by the DINRG, a meeting of the Internet research task force, or IRTF, which meets alongside the IETF. DINRG is interested in preserving the idea of the decentralized Internet, and they met in Madrid at the last IETF meeting. This idea of a fully decentralized Internet is a central tenet a long held belief in the Internet tech community that the protocol designs and the whole idea of the Internet was aimed at removing centralized, monopolistic control of digital communications. Free open protocol designs were believed to encourage rapid competition and enable widespread use of systems by anyone to talk to, well, everyone. The idea of open technical specs and protocols hasn't actually matched the emerging reality of what the modern Internet is. The modern Internet is highly centralized. It reflects at best one or two infrastructure providers for things like mobile telephony, with a skin of competition on top. And there are really only four or five major worldwide data center and cloud providers who are doing all the heavy lifting for almost everything we do online. Geoff asks, what's going on. How did we get here, and was this actually inevitable? Geoff, welcome to ping. What have you got for us this time? Geoff Huston 2:42 Well, George, we're just doing this talk the week after the IETF meeting in Madrid, around that time, anyway, so we've just had a meeting of the Internet Engineering Task Force in Madrid, and there are a couple of topics there which interested me. And one of the ones that I thought might be a good thing to talk about is actually one of the research groups that met there the centralized Internet architecture, I think it's called DINRG. George Michaelson 3:09 These research groups are sort of part of IETF, but they actually, strictly speaking, they aren't the same as an IETF Working Group. Are they? Geoff Huston 3:19 Well actually they are not but they reflect, I think, the genesis of the Internet, which was strongly associated with the academic and research effort, particularly in the United States, initially, but then, as it sort of broadened out, this kind of use of packet switching technology for general purpose networking captured a huge amount of research interest, George Michaelson 3:41 it poses many questions, doesn't it? Geoff Huston 3:44 Well, it does, and it continues to do so. And the IETF, rather than sort of glomming on to the sort of commercial players involved in the production of technology, kept strong links into the academic and research world, very strong links, and they actually formed a separate group that sits slightly in parallel to the conventional production of standards, called the Internet research theory group, the I, let's call it IRSG, George Michaelson 4:18 it's the IRTF, the Internet research task force, but it's not tasked with anything, and it's not exactly a force, is it? Geoff Huston 4:26 It's none of the above. It's like Lord Privy Seal, which arguably is none of the above either. This one's none of the above, but it is a place for academics and researchers to kind of hold structured meetings, exchange ideas, and indeed, from time to time, not only generate Internet drafts to talk about, but it has been known to produce some kind of standards documents, be they experimental or similar to kind of seed into the more traditional, you know, elements of the IETF. One of the classic examples of that is quite recent actually the congestion control Research Group, which has been noodling various ideas on, you know, can we improve on TCP? What are the elements of a good congestion control algorithm, and has been feeding off ideas into the IETF congestion control Working Group as standards documents. So, you know, it's a symbiotic relationship in many ways. And I think it's, it's very productive one for the IETF kind of works. George Michaelson 5:26 Yep. So you were in the din, RG... Geoff Huston 5:31 Right. I was in this, this research group on decentralized Internet. And, you know, you have to kind of wander back a little bit and think of what the aspirations were for the Internet and then see how they panned out to understand the context of this particular work. [George: Yeah], you see, the whole idea of the Internet was that it was meant to be a very, very open technology, extremely open standards were freely available. You didn't have to pay for the paper. Anyone could produce technology based on those standard specifications. And the IETF not only produced that, but also went out of its way to assure you that implementations based on these standards would interoperate with other implementations based on the same specification. [George: Yeah], you can see, I guess, where this is leading. George Michaelson 6:27 We've talked about this several times, and the idea here that you or I could stand up a service. We could get this document, if we were smart enough to know how to code, we could code our own version. We could run these things, and we could talk to other people running these things, and we had a realistic prospect of actually exchanging and talking with each other. That's been like a founding principle that people hold very dear to their hearts, but you've actually made the point that wasn't really the main impetus here. If you really get down to the nitty gritty of why people wanted this kind of investment in openly available standards. It was about competitive pressures and pricing. It was about provision of service in that manner, and the equity of having access to these standards. It was all about pressure on economics, wasn't it? Geoff Huston 7:15 Well, I think there was a very strong economic push behind all this. By making it open, they're implicitly, quite deliberately, building a competitive environment, making the technology not as secret or, you know, the unique property of Company A or company B, but trying to make sure that anyone who wants to compete with the incumbents can enter the market and compete and technically on equal terms. There's no secret sauce going on here that if you take the RFCs and produce your own, you know, Internet browser or application or something else, it'll interoperate. [George: Yeah], you can work with everyone else. And quite frankly, this was taken up, oddly enough, by the public regulatory area extremely enthusiastically in the 90s, I think the 90s and 2000s you see, there was a lot, there was a lot of disenchantment with the monopoly telco model at the time. [George: Yeah], they had turned from being useful public utilities into gouging monopolists at a national level. And, you know, the relationship with almost everyone had been wildly abused many times over. And these were rich companies just sat on technology, sat on a monopoly, [George: yeah], and, you know, extracted cash from from you and I as mere consumers of the phone service. George Michaelson 8:41 Yep, they didn't have very strong motivations to innovate and deliver new services, because they were quite happy milking customers to provide all the things that they knew and loved very expensively in very high standard copper lines. We're doing okay. Thank you very much. Geoff Huston 8:58 Absolutely. And I think there was certainly some degree of, I don't know, resignation that governments could not regulate their way out of this mess that I like in the United States. It took an anti trust action, which has been used only sparingly in the United States and successfully used extremely sparingly, and they did one against AT & T in the 1980s I think was the first successful one since American Tobacco in 1912 if I recall rightly. You know, they didn't play it very often, but AT & T had basically abused, [George: yeah], long suffering American public to the extent that an antitrust action was actually a goer and a gentleman judge Green was put in charge of the divestiture of AT & T and the whole idea was a bizarre thing of dismantling its size, dismantling its unique monopoly, allowing other folk to compete in long distance, notably MCI, which has its own story, but allowing competition in long distance, but recognizing that the local access loops were an economic nightmare. George Michaelson 10:15 Oh gosh yes, Geoff Huston 10:16 they were just a mess. So we actually took at and T's local access loops, and created, I think there were nine, what we call the baby bells, who had a geographic monopoly across each of the regions of the United States, all overseen by Judge green. Now, in long distance, the mantra was, I don't really need to look closely at this competition. We'll fix all ills. George Michaelson 10:41 Yeah. Do you remember, Geoff, there's a story you told me when we were traveling once about a quirk in how that long distance thing worked. Because what typically happened, and we are back in the Stone Age of telephones, is that you would use the phone somewhere and have to pick your long distance carrier. And if you were in a hotel, you didn't want to necessarily use the carrier the hotel nominated. You had to have a way of saying, Oh, no, I want to use this long distance carrier. And one of those mechanisms was that you spoke to an operator and told them the name of your preferred long distance carrier. Now you told me a story about someone who found a way to game that market. Do you remember that story? The point of that one was that they registered the company name anyone, and if a customer rang the operator, and the operator said, which long distance carrier would you like to use if they said anyone that was the carrier they got. Geoff Huston 11:45 It's brilliant. I do remember the automatic way of doing it. The telephone calling cards in hotel rooms became longer and longer as the various previous George Michaelson 11:54 the number of digits you had to dial to specify your preferred carrier Geoff Huston 11:59 To select sprint, to select MCI and so on. And in some ways, there was competition of a sort. But quite frankly, AT & T was still sitting in a pretty privileged position, as it turned out, though, this was not about competition in telephony, despite all of the appearances that it was. It really was, was a competition between the telephone and computer data. And what deregulation in America, and indeed all the other markets that deregulated actually made was the ability of, well, what became Internet service providers to come up and compete against the incumbent telephones, George Michaelson 12:36 the disruptive effect of allowing a new kind of technology that was built around the model of packetized data, rather than traditional telephony models of constant voice, even though the voice had started to be digitized, the idea that you could actually give people the power to address packets and send packets for themselves, rather than having it done on their behalf behind the scenes by the network. That was transformative. Geoff Huston 13:02 It was transformative. It was incredibly efficient in if you look the information load versus your presence on the wire, you know, it takes a huge amount of effort to actually keep the integrity of human speaking. You've got to keep the time signal running. You've got a relatively wide frequency band. You can't stutter the packets. You can't muck around with them to make a continuous flow of analog signal at the other end, you need a lot of wire now, computer data doesn't need that synchronicity. You just jam the packets in where they fit. And you know, in the early days of Internet working, we're amazed at exactly how much data you could cram into a single voice circuit. And so the equivalent of one person speaking, which, even if you speak very, very fast at tops 50 words a minute, whereas if you take a constant rate, 64 kilobits per second. You know, that's a lot of data. That's a lot of data George Michaelson 14:04 when you think that humans effectively live their lives in eight hour chunks, because nobody is actually doing things 24 by seven. There's also a very large amount of downtime in a global network where digital data is quite happy to keep running all day, every day, day and night, Geoff Huston 14:22 right? So the computers can fill the wire very efficiently. So in some ways, the real beauty of packetized data in the Internet at the time was it was just cheaper. It was phenomenally cheaper. So you kind of had these two tenets, if you will, of this revolutionary change in the environment. One was a shift away from switching time and analog voice signals into switching data packets, which made the entire thing cheaper per packet, per quantum of information, and secondly, encouraged by folk like the IETF the report. Replacement of the central monopoly, that not only had a monopoly on the service, but had a monopoly on the technology, and replaced that with a huge number of competitive players who each had access to the same technology base, you know. So this whole thing was meant to, oddly enough, self regulate. George Michaelson 15:23 Here we are on the History Channel podcast, and of course, we're now living in this nirvana of the world of completely liberalized, open access, freely available technology standards. Another short podcast. See you next month. Geoff Huston 15:36 All done. You know, let's stop this now. You and I know it didn't play out like that. George Michaelson 15:43 No, it's the world we are living in has just gone to a slightly different place, hasn't it? Geoff Huston 15:49 It couldn't be more different. I've read somewhere that Google's search engine now has 92 93% market penetration just Google alone. So if you believe what you read in search engines, whatever Google tells you is true, the number of folk who host domain names is really only four. The number of folk who host web content five or six. George Michaelson 16:12 And when we say host here, we mean provide the technological underpinnings. It doesn't matter what the apparent label on the website is who is actually running the machinery, the computing and network infrastructure that lies underneath. [George: Right] We've gone from an infinity soup of people doing this to four or five significant players, Geoff Huston 16:33 four or five, and sometimes the market is really just one. And you know, where's competition in all that. And the kind of question is, how did it go so wrong? And this is where this decentralized Internet research group kind of reared its head, because there is this one theory that says we designed it without keeping decentralization in mind, [George: whoa] that we weren't sufficiently aware of the perils of being taken over by a monopoly or two, and that maybe it's not too late. Maybe we can restructure this technology and put some new kind of guidelines for the IETF and others making technology standards that try and reduce the propensity for centralization. George Michaelson 17:26 So a somewhat optimistic perspective, I would say, Geoff, a belief that it isn't too late firstly, and that we can change our guard rails, change our model, change our ideas about development of technology specifications, and that somehow we can reverse course here. But whilst you are many things, you are not a Dreamy Idealist. Geoff Huston 17:49 I'm not a fantasist, no, and that one is pure fantasy. So I'd like to sort of, I suppose, roll things back a bit and then come at this again and try and illustrate the fact that, quite frankly, it couldn't have happened any other way that this this was inevitable. It wasn't that you and I and anyone else around at the time had a lack of foresight or wasn't thinking hard enough about the evils of centrality and monopolies. It didn't matter what we did, this was always going to happen. George Michaelson 18:20 So Geoff set the scene. Tell us a story, Geoff Huston 18:23 I will. And the first thing is, there is no network in the Internet today. None, George Michaelson 18:30 that's a very strong statement. Geoff Huston 18:32 It's a really strong statement. But you know, over the last 10 to 15 years, there's been a quiet, but extremely subtle change in the Internet. Almost all of the content that you and I and every other one of those 6 billion or whoever many users there are the Internet, do not drag our packets wearily across the entire globe, you know, haul them down submarine fiber cables and so on, on the entire little way to get from the server on the distant part of you, know, the other side of the world, to you and I most of the time, most of the time, that doesn't happen. George Michaelson 19:11 So the general image that we carry of the network as some sort of spider web or mesh of connectivity between things you're not saying it doesn't exist. You're saying, transactionally, the things that we do don't, as far as we are concerned, exploit any part of that, except for a very thin skin on the surface. Your packets don't randomly go through this mesh to arrive at someone else's door. They're going to places nearby. Geoff Huston 19:42 Well, you know, like many folk, I watch streaming video on the Internet. That's a huge amount of volume. And streaming video for me is served in my own town, a different town from yours. Yours is served in your town, even services that you might think come from somewhere else, like my electronic mail services, my document storage, a whole bunch of other cloud services tend to seamlessly and invisibly follow me wherever I go. And so what we've actually managed to do is rather than have the network get services that are a long, long way away and connect them to me. We've been able to drag those services close to where I am all the time. George Michaelson 20:29 The network exists, but the network doesn't exist so that my packets get things from far away. The Network exists so that I get packets from as fast and as near to me as I can, and other things use "A" network to make that work, but what that other network is isn't necessarily the public Internet. Is that where you're heading, Geoff Huston 20:55 that's exactly it. We've got so much abundance in computing, in processing, so much abundance in storage. You know, this stuff is cheap and plentiful that running a just in time delivery model says, Ah, you want this file from over there somewhere in I don't know, Europe, Japan, whatever. I'll fetch it for you, because you've asked Hang on a second, I'll just set the packets in motion and set the transaction up. Instead of that, we're doing it just in time. A number of people have asked for that file. I think I'll store it in a local cache right under your nose, so that when you ask for it, I'll go here done. That implies a completely different model of what I would have considered to be the cost component of why things are done. I mean, when you start this journey, the idea that I can afford to just have the entire corpus of books and films and TV and radio and music at my door, that's ludicrous, because that's petabytes of data, and surely, economically, the most efficient way for me to get that is for it to be held reliably somewhere, and for me to come to that place over a network that will supply anything. And you're telling me that model is completely wrong, and the economics here don't work that way. Well, I am a bit dumbfounded. Well, for one, we humans are a bit of a herd mentality. We all do the same thing at the same time, and to some extent, I only need to be able to get 10% of all the world's stuff close to you, and I've satisfied your every possible whim, because no one's phrased that far away. Look at something like Netflix. The entire catalogs that are in demand in that locale are stored in that locale. Look at Google's Gmail. That mail is close to you. It's not on the other side of the world. Otherwise you would be dying waiting so long for something to respond. So we're actually able to use these networks and the associated caches to place the services that you and I use under our noses at the other end of the access network. George Michaelson 23:11 I don't see how it's possible to do that build out close to me without spending an astronomical amount of money to build expensive boxes. Close to me, it just feels like a lot of money's been spent here. Geoff Huston 23:28 The boxes are cheap. They've got the expense. The expense is keeping the air in the room that holds the boxes to a cool enough temperature the data centers. But, you know, we've been building data centers like crazy, and we continue to build, because of AI, even bigger data centers, even crazier. And we have actually not built phenomenal amounts more network. We've built phenomenal amounts of local cache storage. We've changed the entire model of provisioning in this sort of digital world to duplicating the services, duplicating the data and putting it close to where markets matter, so we can deliver it immediately. Just in case you ask, why do we do that? Because in networking, cost is distance squared. I can cut the distance and just simply track it across a few kilometers. It's fast, it's cheap, it's astonishingly good. And you know, why do I need 6g that's the current buzzword, mobile networks, in the access network, because the content that's being pumped across that last mile access network is basically at the head end in your area, pumping it directly from the data center into that access network. You're not hauling it through a whole bunch of transit networks. You just don't do that anymore. George Michaelson 24:56 So models that we brought to the table when we were at the entry point of building this network, they were very predicated on big, expensive computers, big expensive disk farms, big data centers with big power budgets. Geoff Huston 25:10 Yep, scarcity, George Michaelson 25:11 cheap networks to get to the door. And you're saying what's actually emerged is that networks remain cheap, but in some ways, have become the most expensive component, purely and simply because of the time it takes to move packets in the real world across the surface, whereas the cost of computing, the cost of storage, has gone down enough that you might as well just put it everywhere and put on it things that people want to see as close to people as possible. That implies an astronomical reduction in the cost of computing and storage. I mean, like, unbelievably cheap compared to what it used to be. Seriously, Geoff Huston 25:51 how much does it cost to stream a high definition 4k movie to you now in 1990 if you'd asked that question, I'd be sitting on the calculator, punching zeros for half a day. You know, that's a lot of data you're trying to push across the world's networks. You're elbowing other people out of the way. You know, that's going to cost you serious bikkies. How much does it cost to drop a movie, a streaming video for 90 minutes across a access network in today's world? Fractions of a cent? George Michaelson 26:23 Well, I don't think anyone's charging. Geoff Huston 26:23 Well, they can't anymore, right? You're in a world of abundance, and it's a different kind of world. So what we've done is we've built out those edge networks five into the home 5g in the mobiles. You know, we've absolutely built them out, and that massive capacity has actually enabled us to bring up commercial models for streaming high volume content, particularly video, but yeah, very high volume streaming content, which in turn, kind of keeps on fueling the demand for even more access capacity in the access networks. All that gives you is these phenomenal economies of scale. So because everyone's doing it, it becomes cheaper. It's no longer customized. It's just what everyone does. Stamp it out in the millions and just deploy it out. The unit cost of carriage and edge networks is plummeting right [George: right] -Now how did that happen? Did we plan for this? Was this something we knew about 40 years ago and said, this is the direction we go? Course not. You see, the one thing we did do when we de regulated is that we moved into a market based economy, not a command and control economy. Now one assumption was that would create competition, the market would self regulate because of competitive pressure. But the other part, a market based economy is not based on what I want to sell you, it's based on what you want to buy. And so that second bit is actually where we've managed to totally succeed. George Michaelson 27:58 So that first bit of competitive providers of service. I feel like that's kind of arrived for me in my economy. I mean, I have an element of service delivery that's a common national fabric, that's the National Broadband Network here, and there are comparable systems in other economies from one or two providers. And then on top of that, I'm picking and choosing amongst a very large number of competitive RSPs, higher layer providers who sell me my Internet and change aspects of their cost model behind the scenes for how packets flow internationally. Same with telephony. I've got an esim in my phone. I'm free to choose another provider where I am, and flip the bits and magically get my 5g or 6g from another provider. So you haven't yet sold me on everything about what I'm doing has been denuded of any competitive tension. It feels like there's a bit of competition and non centrality there. Geoff Huston 28:58 Why would put the competition at the level of a veneer. It's really a competition in billing systems. [George: Ooh] -The underlying massive billion dollar investment, whether it was in fiber infrastructure that connected every house, and, you know, don't forget, every residential unit cost a couple of $1,000 in whatever your local currency might be, to connect it up. And no one's arguing that you can make that competitive. You can't screen 20 fibres down the street. You know, one blue one. George Michaelson 29:28 No. It's two at best. It's a duopoly Geoff Huston 29:30 No, it's 1g POD Network, no matter where you sit. That's the only way it cuts it. And even in the mobile world, you might think there's a lot of mobile providers, but if you cut through it. The spectrum, no matter you know what's going on, typically allows at most three, just three, everyone else is virtual. George Michaelson 29:49 That original US decision to take AT & T and split it up into the baby bells, the RBOCs. In some senses, that's the persisting model of the world that we live in. You're saying there is. Layer where there is simply no room for competition and it just has to be there can only be one or maybe two or three. That's how it is, Geoff Huston 30:08 right? It's kind of a crowding out theory that the capital costs are high enough and the return rates aren't brilliant. Burying fiber in the ground is not going to make you a lot of money, no matter how you cut it and varying. You know, 20 different companies fiber in the same piece of ground is a recipe for a disaster, not a highly efficient infrastructure. So bottom of this, there is actually not a highly competitive investment infrastructure for these kinds of networks. George Michaelson 30:37 So at the bottom, at the bottom, okay, I think I might have to buy that story. That feels true, but you have been leading us to a point that says there's actually less competition than you think, all the way up to the top. Geoff Huston 30:51 Yep. So the next element of this, you know, in computing, I think there were only two innovations, the first, Bell Labs, 1947 the solid state transistor stunning, because the entire thing about valves was that they were inefficient hot. You couldn't build a million of them in a room. It just wasn't a scalable technology. Whereas the transistor was totally revolutionary. Didn't run hot, didn't require huge amounts of voltage. You could jam them together, amazing stuff. And the world you know, transistor radios appeared almost immediately, and it made companies like Fairchild overnight, because it was just so much easier than valves. What an invention. The second one, not so well lauded, but I think equally fundamental was the invention of the integrated circuit in 1957 I think George Michaelson 31:50 People have been playing with rather weird ideas of a thing they were calling something like tinker toy that was going to be sub miniaturized components, but you still plugged them together on teeny weeny circuit boards and built things out. But what an idea. Geoff Huston 32:06 I think the idea was kind of simple in some ways. I had this transistor built of doped silicon components, you know, positive, negative in a gate. Why don't I put two of them together on the same silicon substrate? If I can do two, I can do 50. If I can do 50, I can do 100 [George: Yeah], and for the last I don't know, since basically 1960 so that makes it now 85 years. We have taken that one idea, integrated circuitry, and pushed it to the point where we're now talking about 1 trillion features on a single silicon chip, George Michaelson 32:46 1,000,000,000,00 Geoff Huston 32:46 1 trillion George Michaelson 32:48 1 trillion. Geoff Huston 32:49 Yeah, the feature size is now dropping just around three nanometers down into two nanometer so we're now to 1 trillion gates around a three to two to three nanometer feature size is the current absolute state of the art. And of course, the thing is, the cost per transistor has plummeted the same way, because, in many ways, the fabrication process, modulo some arithmetic, is basically a function of the wafer, not the function of the number of features on it. Now that's a pretty gross implication, but nevertheless, the cost of computing has plummeted in line with that phenomenal change in Moore's law. George Michaelson 33:31 I would say at this point Geoff that the cost has not plummeted to the point where I am showered with a new Gigahertz CPU every month for free in the cereal. They Are still extracting rent from the ability to give me these phenomenal things. Yes, but that watch you just held up to the steam radio that no one else can see. You didn't get that for free. Did you? Geoff Huston 33:53 No, but I paid a whole lot less than the super computer that my university bought in 1985 which had less computing power and less storage than that, George Michaelson 34:03 not a trillion times less, Geoff Huston 34:05 but cheaper, cheaper, George Michaelson 34:07 I'll give you cheaper. So this engine room, there's a shorthand to describe this, right? I mean, this is Moore's law that you're talking about, Geoff Huston 34:16 right? This is Gordon Moore, who worked for Intel. He led Intel at the time, who observed, by I think it was the 70s, that the integrated circuit industry was actually being able to achieve reducing the feature size continuously, such that the number of transistors per, I don't know, square centimeter on a silicon chip, the number of transistors doubled every 18 months. That what was going on here was that computing became more abundant continuously, and the cost of computing, the cost of unit of computing, came down comparably. So tomorrow's chips might cost you the same in dollar terms, but they did twice as much work. Now that wasn't just a one off. That was every 18 months. Things changed now they had a profound impact on the computing industry, because in five years time. The computers that are there in five years time are more than four times faster, but cost in dollar terms less. So any business plan you had, I'm going to buy some computers and set up a business. This was in five years time, what you're running is incredibly expensive, slow and won't work anymore. You've got to get rid of it because you have to change the model. George Michaelson 35:48 So if you consider classic commercial production of real goods and services, you buy a biscuit factory, and you get these hokey machines that stamp out biscuits, and they always seem to be covered with cream colored enamels. It looks cleaner, but one of those biscuit stampers, you get 25 years out of it, and you might get longer you're making Arnott's Tim Tams on the same machine your granddad made Tim Tams. You're saying, No, you have to extract all the value you can get out of this computer in probably three years, Max, because someone else will come along and do it cheaper. Geoff Huston 36:26 They'll do you over. They will do you over because Moore's Law says, even if they wait for a year, they will get more computing for less money. And so if you think about it, from planning todebut of the service operation to, you know, consolidation in your market to maturity and then all the way to obsolescence. A market service offering based on a static technology base is about five years to do the lot [George: Right], enter, make money, die in five year cycles. Now, if you think about the way the telephone world worked, and I really like this story, I was told about the bell operator, the bell Bell Laboratories. So we've just been working on this phenomenal thing called asynchronous transfer method ATM, which was going to completely revolutionize time division multiplexing and time division switching. The whole idea was, instead of using an external clock to time which fragment of data belong to which conversation George Michaelson 37:23 and managing those clocks was incredibly complicated. Geoff Huston 37:27 It was difficult and expensive AT&T 5ESS switches cost, you know, a major economy, a lot of money. So instead of that, the idea was you put a micro header on the fragment, and, in essence, the switching units knew by that micro meter which conversation that packet fragment was part of right revolutionary, it would change the entire economics of telephony. So Dave zinkowski, I've been told, presented this to the regional Bill operating companies at one of the Progress meetings, and explained how this was going to completely revolutionize the communications in the telephone industry. And the answer he got back was, you know, Dave, we make money by sweating our equipment for about 25 years, [George: yeah], and then we're ready to replace and, you know, quite frankly, 10 years before that meeting, they'd just done a massive forklift upgrade into getting into digital switch. George Michaelson 38:23 Yeah, you're coming at us saying we need to replace with new tech, and we haven't finished sweating the old yet. Geoff Huston 38:29 We're not ready. We'll go back to the lab, cook it some more, and see us again next year, and then about 10 to 15 years, we'll be ready for you. The folk in the labs were unimpressed. They thought they'd just done the bees knees. You know, they thought they got this Ripper technology, and the Bell operating companies are going our investment money isn't ready for you. We're not able to reinvest. Now, that's a big message. And the funny thing was, they went to Sun Microsystems and said, you have to spend a fortune upgrading from 10 megabit ethernet to 100 meg FDDI and you're running out of grunt. You need a scalable technology that doesn't make this happen every few years. Let's try ATM and odd enough, some went, you know, [George: yeah], yay. The computing industry was running at a different pace to everybody else, right? [George: Yeah], so Moore's law is brutal, absolutely brutal. And what's going on is that, traditionally, communications was the economics of scarcity. It's why we paid for telephone calls. It's why long distance calls are expensive. International calls were hideously expensive because pricing was used as a rationing mechanism. George Michaelson 39:46 When you've made an investment in copper lines to run telephones that you're going to recover on 50 Year commercial bonds, you absolutely need to make a lot of money on that for 50 years, Geoff Huston 39:58 50 years. So they're based on. In pricing as a means of rationing to a scarce resource. What happens in abundance? Well, you end up giving it away because it's so plentiful that everyone has access to it. There's no rationing function left, because this is so much of it. George Michaelson 40:18 So the thing that you sell people can't be the scarce thing anymore, because it's not scarce. You have to sell them something else, Geoff Huston 40:27 right? So a new form of economics comes in, and it's brutal, in the same way that Moore's law is brutal. One of the backers of PL PayPal, not Elon Musk, but someone who hasn't been as notorious in recent years, but has an equal issue, I think, with the way societies tend to work. There's Peter Thiel, right. It came out in an article, I think it was in the New York Times, and one of the quotes of it was, competition is for losers. George Michaelson 40:58 Oh, dear. That's a very provocative statement to make for an old time economic student, isn't it? Geoff Huston 41:04 Competition is for losers. The idea is that if you're going to take a punt on Moore's law, if you're going to pump whole lot of money into believing that a technology will become viable in three years time, that isn't out here now, and you're going to pump a whole lot of money into a service that you're trying to build on the fly because you don't quite know how it's going to work, then the last thing you want is to have someone nipping at your heels. And so you tend to fixate on a different model of venture capital funding. [George: Yeah], that model is when it takes all. Competition is for losers. George Michaelson 41:42 So we're not really here to debate the political good or bad. This is just an observation of a reality. If you've moved from long term capital investment, sweat the asset for 50 years to I have to spend money and get it back inside three years or I'm dead. You don't want to share that three year window with anybody. Geoff Huston 42:03 Exactly. You don't want to share. And so this kind of high risk very, very unstable environment, thank you Moore's law, and it is very, very unstable, tends to mean that the investment money takes on a momentum that says, not interested in competition. Thank you very much. I really don't want to open up this technology and have lots of folk devaluing, you know, my investment into the future. I wish to keep an absolute set of control over this winner. Take all. [George: Yeah], if this one comes off, it's me and me alone. And so if you're thinking the competition is going to regulate, that the actions of the investors are going not at all, and now there's no control. George Michaelson 42:50 So this is sort of a statement that said, while we were making the protocol spec and the technology specs neutral and open and advising towards competitive market services that were going to benefit us all. The Nature of Things changed so that those technologies are all fine and dandy. But the one thing nobody is driving to who's got money to invest in this space is a competitive open market. They might be using the protocol they're not remotely interested in the competitive component in it? Geoff Huston 43:21 No, they're trying to minimize, if you will, the risks associated with what was already highly risk investment because of Moore's law, basically. And one of the ways to reduce the number of unknowns is to actually go, I'm not interested in competition, just not. And there are a number of ways that you can express that interest. One is huge amounts of secrecy. You notice, the current big players say extremely little about what they're doing, how much they know and how they do their business. Equally, also, you're prepared to buy anything that looks like future competition. And there was a secondary industry, I think there still is, where entities spin up not to actually have a market impact. That's not the name of the game. They spin up only to be bought by one of the emerging or the emerged giants. That's their entire, their entire mode and purpose of existence, to be bought to be perceived as a potential future threat that needs to be neutralized before they hit any moment. George Michaelson 44:22 We used to say that for any system you could imagine buying in the open marketing, computing, IBM Zurich Labs had probably built one and had a version of it in storage that they could wheel out if they felt they needed. In some ways, what you're saying is a modern version of that story, but instead of it being IBM at scale, making things, you're saying, it's venture capital at scale, buying things in order to make sure what they already have can continue to make money, Geoff Huston 44:52 Right? Because once you've actually established on a winner take all basis, a degree of dominance in the current environment. So in Moore's law, your biggest enemy is the future. Your biggest enemy is, you know, cheaper, cheaper processing, cheap, cheap, cheaper, cheaper storage. And so to neutralize that, you have to make sure that potential competitors never gain momentum. And for that, you need to be the largest of the large because you're going to wipe everyone out that's even close to looking like they're serious competition, not now, but in the future. So what you're building is an assured outcome almost a decade down the line, because everything else in down the line, decade with Moore's law will not be the same as today. So if you want to survive, you need to make sure that no one else is going to jump up and take over your market on you. Small doesn't work. Nimble doesn't work. George Michaelson 45:51 All these iconic ideas we have about new technology deploying in small, fast, nimble, revolutionary start up companies, you're saying, what's going to happen is they're going to be acquired by someone who's big and converted, if they emerge at all, into something that works at Big so that no one else can occupy that space at scale Geoff Huston 46:14 Exactly. And so this is now a scale argument. And so if you look at that point of view, it didn't really matter what the technology was. It was always going to be strongly dominated by scale players. Let me take an example. This is a bit of a breathing space security, and there are a couple of models of security that are almost antithetical. One of them sounds highly, highly, highly centralized. That's the model used by the DNS in its security framework. There is a single trust point, a single key pair, which all other security sort of credentials in the DNS ultimately rely on. So it's a single, rooted hierarchy. Now we've gone to some degree of pain and effort to make sure that that single key is not controlled by a for profit, private sector money grubbing entity. You can call, you know, the good folk at PTI or IANA, many words, but you couldn't call them that. They're not no George Michaelson 47:24 People like me go and participate in somewhat silly ritualized behavior in secure facilities to guarantee a level of public exposure to this being done for the common good in a fairly open manner. Here's the goods coming out the door. If you want there to be one key to rule them all. It's made in this way. Watch it and enjoy Geoff Huston 47:46 So in theory, a single trust point could be highly centralized, could be prone to abuse by monopolies, but an extraordinary amount of effort has been been taken to cut that out. George Michaelson 47:59 That's one model, but you're saying that there's another model that is in tension? Geoff Huston 48:03 Well, hang on, a second degree of market acceptance of DNSSEC George Michaelson 48:07 you're measuring that, aren't you? Geoff Huston 48:09 Yeah, there's very little less than 3% of the queries that we see out of the major recursive resolver refer to names that are being DNSSEC signed. Folks don't use it. George Michaelson 48:21 So we built this model, and it has attractive, socialized human equity behaviors. Geoff Huston 48:27 And even though it is abusable in terms of centrality, we took extraordinary measures to negate it. We kind of sucked all the motivation out from the venture capitalists, another way of putting it. And yes, no one uses it [George Right], and it's unlikely to have its prospects change interesting and the other form. Now I contrast that with the other one, which is oddly enough, started as incredibly decentralized. It's the web PKI, the web keying infrastructure, what we use to make HTTPS, colon, slash, slash, actually work, because right from the outset, there was no single entity that issued certificates for domain names. There was a forum between browsers and folk who do this work, certificate CA, certificate issuers, certificate authorities. And in essence, you meet some quite stringent criteria, but then you can mint certificates. So there's not one, it's not two. Currently, I think the number's up around 900 George Michaelson 49:32 I have serious concerns with aspects of public oversight there. So yes, to all intents and purposes, it's a wonderful framework where, if you are high enough in your capital, investment and commitment, you can enter this framework. But you know, and I know, that there are also bad actors who nestle in here, state actors who nestle in here. There are all kinds of things going on in that space that make me feel extremely uncomfortable. and wouldn't you know it, I don't actually, realistically get to pick and choose. Every time I get a browser update, I get a new one added. Geoff Huston 50:09 The point I'm going to make is that it's not decentralized, what it might sound 900 players out there being able to mint certificates, that you'd have a panoply of choice, and that everyone would have their opportunity to make their impression on the world in a strongly competitive environment. As a certificate authority, Let's Encrypt mint certificates for about 60 to 70% of the market, and rising George Michaelson 50:35 For free. Geoff Huston 50:38 That was the key, but one player dominates this space. Now you might say, well, they're doing it for free, and they're not real. The answer was, free as a fine free as a fine alternative. Look at Cloudflare and their presence in the Content Distribution Network business. Their base offering is free. You can enter a market by being disruptive. But the issue is, if it's highly decentralized and highly competitive, there should be other ways to counter that. But what we've found, oddly enough, is even in a mechanism that is strongly, strongly designed to be decentralized. One single player coming up with basically a radical, different model. I'm doing this for free took over the market, and in essence, everyone else is forced to compete with free, and they're not having a good time because free is so hard to compete George Michaelson 51:30 with. I mean, I used, Let's Encrypt. You use Let's Encrypt. Geoff Huston 51:34 We all do. It's hard to ignore free. George Michaelson 51:36 We have kind of ignored a governance question and a social equity question, and all kinds of considerations about why we trust what Let's Encrypt are doing. So it's not that there aren't problems here Geoff that we would need to discuss, but at one level, I think what you've said is, if you want to take on air quotes, bad, evil, centralization, one of the ways that you do it is by offering air quotes good, beneficial centralization. Geoff Huston 52:06 I'm sorry I am not putting, Let's Encrypt, up on any pedestal. George Michaelson 52:10 You're saying they are central. Geoff Huston 52:12 I'm not saying that they're good or bad. I'm refraining from that. What I am saying that is about as good a definition as you find of economic dominance in a market the standard, perfect dual Hirsch, later, the HH index would go off the scale, going, dong, dong, dong. This is a strong, monopolistic market right now. It is dominated by a single player. There is not a realistic amount of choice going on out there that consumers can avail themselves out there. It is a warped market, twisted by centrality. So my point is that sometimes systems that we design to be, if you will, by very design, highly centralized, we actually manage to diffuse like DNSSEC and make that less of an issue. But oddly enough, it's been at the price of commercial success. Whoops, on the other hand, one that was designed from the outset to be resistant to centralization, that you know, was an open one that anyone could join and sort of Whoa, taken over by Let's Encrypt, taken over by one player, and it is an effective takeover. Now, whether their hearts are pure or not, doesn't enter into this discussion, so I'm not impugning their motives or anything else, but I am saying they're dominant. They're Central. George Michaelson 53:33 How was this presentation received? In the DINRG Geoff?, because it sounds like you said, DIN isn't happening. Geoff Huston 53:45 Oh, I've long since, and I think that's why the chairs invite me to present long since, you know, resolve myself as being, you know, the Contra voice out there, singing in the eye of a harmony, you know, saying that you can't build and design networks in isolation of the economics. And in some ways, the economics is a much more powerful motivation than the technological aspects. And when you're in a space that is radically changing, such that each generation of technology is almost unrecognizable, generation by generation. Today's network dominated by local caching, dominated by, you know, content distribution networks, dominated by local access loops and feeder networks that aren't necessarily public, is not the network that we operated in 2005 20 years ago. It's a different network, right? And I suppose where I'm going is that change, in this case, true change was driven by the economic windows opened by the continual refinement of Moore's law. Now, if we ever reach the end of that path, and I've just read a book about AMSL, and I'm not sure we're reaching the end of the path anymore. There are still some strong hits from the clever boffins to make it better. George Michaelson 55:06 But if we do Geoff Huston 55:08 Oh If we do, it's a new story, because as soon as the music stops and things go stable, you're kind of left with whatever mess we have, and that will be very difficult to upset, even with the dint of technological refinement. But right now, where we are is this constant upsetting of the apple cart every few years, causing the major players to effectively try and put their stamp on everything to be phenomenally dominant as an artifact of the ecosystem, the economic system, not as an artifact of the technology itself. So as we move into a whole new world of network, moving away from IP addressing, moving away from destination based, hop by hop forwarding, moving away from all this peering in transit, into a completely different world of service names, content distribution and name based mechanisms. You know, it's not clear that that will resist the aspects of centrality that I think are inevitable while things still change. George Michaelson 56:10 Perhaps we might have to actually come to understand that centrality as an architecture, as an innate feature of things done at economies of scale worldwide is, in itself, sort of not necessarily wrong. I don't mean good or bad, but I mean it's a good fit for the class of problem we're trying to solve. Geoff Huston 56:33 I think it's an excellent fit for the class of problem. But let me get into the moral dimension and say predominantly, it's always evil, predominantly, predominantly, abuse is always going to happen when you're so dominant. And the struggles that I think every single sovereign nation finds when trying to understand the overwhelming influence of these digital behemoths and the destruction of things in their wake, be it, you know, the advertising networks, newspapers, journalism, you know, a whole number of social institutions that we tended to rely on have been completely destroyed, perhaps irrevocably, and nothing built in their place. And I personally think no matter how you cut it, that's evil. It's evil from the health of a society's perspective, and who has the ability and motivation to build up again in that way? [George: Yeah], it's not clear, and that's really the evil side of this. George Michaelson 57:32 This is not fixable by inventing new technology or thinking like technologists, because these are not fundamentally technological problems. Are they? Geoff Huston 57:43 Right? So, you know, I share in some ways, the motivation from the folk about, you know, decentralized networking, that centrality in the long run is a really, really bad thing, and from that kind of dimension, absolutely. But you know, the fault is not in the way we cut this technology. It's not in the design of the Internet. The fault is actually in the way in which this economic system works that has created, if you will, a perfect storm for money to aggregate into ever larger piles and have ever greater influence. And our ability to counter that is something that I don't see is happening anywhere in the world. So that's, if you will, the tough question. So bring on the economists. Bring on, keep the technologists out of the room. This is more a problem around economics than it is around technology. George Michaelson 58:36 Yeah, a hard lesson to learn, Geoff, but it's interesting times we're living in when we come to understand the nature of the beast, that's really great. Thank you. Geoff Huston 58:45 That's been a pleasure, George, thank you. George Michaelson 58:46 And is there somewhere online people can read more about this is your DINRG talk floated into the net? Geoff Huston 58:53 You'll find various things about decentralization on potaroo, dub, dub, dub, potaroo.net including the presentation, which talks about, you know, the Impact of Moore's Law on centrality, and also some essays around trying to cope with big, particularly the American experience, going back to the Gilded Age of the 1890s that this is not a new fight. Big has always been a problem, and the various interests about we can just regulate our way out of this through to No, no. Sometimes commercial entities just get so big you need to take out a hatchet, irrespective of whether they're doing good or bad, they're just too big. And that argument, that argument, has been raging for decades. George Michaelson 59:38 We'll make sure there are pointers to these stories in the blog that goes along with the podcast. That's great. Geoff, thanks, Geoff Huston 59:44 Great. Thank you. George Michaelson 59:47 If you've got a story or research to share here on ping, why not get in contact by email to ping@apnic.net or via the APNIC social media channels. Also remember. But the measurement@apnic.net mailing list on orbit is there to discuss and share relevant collaborative opportunities, grants and funding opportunities, jobs and graduate placings, or to seek feedback from the community on your own measurement projects. Be sure to check out the APNIC website for all your resource and community needs until next time you.