Geoff Huston - 00:00-00:16 That money which was fueling that industry is now being diverted into what, AI? And it's not just that industry, because that's nowhere near, nowhere near the trillions of dollars we're going to need. Nowhere near the 20% of GDP that we're after. This is now big. And the real issue is, at what point And how does this become viable? And it's from an individual consumer's perspective. What are you going to give up? What are you going to forego? George Michaelson - 00:39-00:48 You're listening to PING A podcast by APNIC discussing all things related to measuring the Internet. I'm your host, George Michelson. This time, I'm talking again to Geoff Huston from APNIC Labs in his regular monthly spot on ping. Geoff and I discussed data centers, not their routing architecture or the issues in data transport, but the economics and the social costs of a decision to aggressively deploy artificial intelligence. By its very nature, the large language models and generative pre-trained transformers of AI, that's the LLM and GPT systems demand massive investment in computing systems to perform their data analysis and construct the models which are applied to the problems we give them. You will have heard of models with names like 35B or 100B or 5B. That B stands for billion, the number of distinct parameters or weights which have been calculated from the data sources This is far bigger than the number which might actually be used for any specific problem. Reductions over this parent model can be turned into something you could run in smaller configurations inside your own computer. even at tolerable speed if you have the right kind of GPU card and enough memory. Or you could run them in the cloud, in a data service that you rent from a cloud service provider. GPUs used to be exclusively graphic processing engines, typically used for gameplay in super-realistic, immersive rendered worlds. But GPUs are massive parallel computing systems in their own right. They perform a reduced set of operations very quickly over data in high speed memory on the same card as the GPU chips. The same basic idea allows the massive LLM and GPT parent models to perform their initial work and maintain state against all the information being fed in their GPUs. None of this comes for free If you have a gaming PC with a modern GPU, you'll be well aware of how hot it runs, how much power it draws from your PC power supply, and how much cooling that demands. Now, multiply this by thousands, or even tens of thousands. Think about the infrastructure that has to be built to run something at this scale 24 hours a day. Making a data center has never been cheap, but we have kicked off a race to build out, to consume the land, construct cooling and power sources all in order to stake a claim in the physical and digital landscape so that tech companies can own market share. This has quite exceptional effects on the landscape and on the national economy. What's really going on here? Have we thought through all the consequences of diverting this much finance and construction input and technology to a battle between hyperscalar tech companies. Geoff, welcome back to Ping What should we talk about this time? Geoff Huston - 04:01-04:15 Hi George, look it's good to be back. I've taken some time not to talk about it, but it's it's time I let you and our dear listeners in on a little secret. I think I'm going to build my own AI data center. George Michaelson - 04:15-04:16 Oh no. Geoff Huston - 04:17-04:23 Oh yes. Oh yes. It's time. It's time we talked about this secret obsession of mine. I'm going to do it. George Michaelson - 04:23-04:29 You know, I can say on the upside, at least we're not talking about the DNS today. Geoff Huston - 04:29-04:49 Absolutely not. We're going off and catching the rocket ship to Pluto. Because I've been assembling a little list of the things that I need to actually build one of these data centers, and once we've gone through the list, which is just a tiny bit scary, Well then talk about the kind of cost, because again, just a tiny bit scary and what that really means. George Michaelson - 04:49-04:50 Okay. Hit me. Geoff Huston - 04:50-05:01 Right. I don't want no squidgy nonsense data center. I want one that can really do some decent AI, as the current thing is. This is more than artificial intelligence, this is super intelligence. George Michaelson - 05:01-05:07 Oh please, let's not go down that pathway. Let's leave that off the table. Geoff Huston - 05:07-05:15 But what am I going to need? Well, I'm gonna need at least sixty thousand graphics processor units. George Michaelson - 05:15-05:18 Sixty thousand? Sixty thousand. Geoff Huston - 05:18-05:46 Yeah That's six zero zero zero zero. Yes. They're not cheap either. But you know, not just old ones I need them using the latest 3 nanometer silicon fabrication. These things are the best we can produce, because, you know, 60,000 of them. Okay? I need memory. I need fast memory. I need 3D stacked, very high bandwidth memory. I need a lot of memory. George Michaelson - 05:46-05:57 So that's quite apart from the memory that's built into these GPU cards, you need classic computer memory in the rack-mounted computers the GPUs sit in as well. Geoff Huston - 05:57-06:28 In the data center. The GPUs look at memory as an external shared resource and use the connectivity fabric in the data center to actually connect to memory Wow, which means the memory bus is actually a dense mesh of 800 gigabits per second optical systems and extremely fast high-density switches that can switch small memory packets to a sustained rate of 800 gigabits per second. George Michaelson - 06:28-07:09 If we're back in the old days of mainframes and the idea of DMA, direct memory access, which was a way of keeping the CPU out of the loop to let peripheral devices take control of a bus, send data directly into memory, and then return the bus to someone else to use. You're saying the modern analogy in an AI data center is that the GPU needs to access high speed RAM and it has optical interconnects and optical switches and a fabric and the GPU talks directly to this RAM over a switched fiber network. Geoff Huston - 07:09-07:27 No more bus. We're using RDMA. R for remote direct memory access. Which kind of makes you wonder what direct means, but okay, we're using RDMA and let's not reinvent things. We have Ethernet, let's just use it again. And again, let's not reinvent things. George Michaelson - 07:28-07:32 These these are put into Ethernet frames. Geoff Huston - 07:32-08:04 Yes, and we then run UDP over that. So this whole thing Rocky V2 is indirect memory access over Ethernet over IP. Oh by the way, you're not allowed to lose a packet I saw a presentation a few months ago where the impact of a single lost packet was yet to throw away around 12 hours of model generation. because what you thought you'd written into memory wasn't there anymore for just one packet. Not allowed to lose packets. George Michaelson - 08:04-08:18 Sixty thousand GPUs, this memory doesn't just sit in boxes, it has to sit inside computing systems, it has to be connected to controllers. How does this look in density terms? Geoff Huston - 08:18-08:41 Over a hierarchy of switches. Things get very scary. Let's think a little bit about those 60,000 GPUs again. How much kilowatts of power per rack? I'm not going to put one GPU in a rack. That's just crazy the data center would cover a city. So I'm going to stack 'em in and s densely. So I'm going to start consuming, ooh, 200 kilowatts per rack. George Michaelson - 08:41-08:43 Two hundred kilowatts. Geoff Huston - 08:43-08:56 Kilowatts per rack. I thought my house used electricity. No, no, no. This is my suburb or more. This is a lot of power. But that's per rack. And I'm going to need a lot of racks. George Michaelson - 08:55-08:58 How many racks for 60,000? Geoff Huston - 08:58-09:13 Well, 72 GPUs per rack, that's about 3,000 to 8,000 racks. Each one's twenty kilowatts of power that's you know, edging into the gigawatt of power George Michaelson - 09:12-09:14 That's a city of power. Geoff Huston - 09:14-09:28 The City of London only uses in peak peak six gigawatts, and off peak is a whole lot less, whereas these things never sleep. It's a gigawatt every hour Every minute. All the time. George Michaelson - 09:28-09:28 Right. Geoff Huston - 09:28-09:30 This is huge. George Michaelson - 09:30-10:09 Forgive me if I run sideways again, Geoff, but my memory of the days when I lived in a house where somebody had a gamer GPU was that his fan on his computer ran continuously. This thing is not terrifically efficient at converting electricity to computation. It converts electricity to computation and heat. And you just said you put seventy-two of them in one rack and you've got enough racks to get sixty thousand of them, so that's a large machine room of many columns of racks, each one of which is a heater. Geoff Huston - 10:09-10:17 Yes. So I've got a lot of heat I've got to get rid of. And traditionally, because it's cheap we've been using air. Now the thing about air is that it's not very dense. So to get rid of a lot of heat, you need to move a lot of air. a whole lot of air. And in fact, you can't do that without jet engines or more, because to get that much heat dissipated out of those racks, you've got to move the air, probably at the speed of sound or more. That ain't gonna work. So you turn to the thing that we've toyed with for about the last 40 years. I don't know if you remember the very early supercomputer, the Cray One. If you looked hard, you would find some water pipes going through the bottom of this damn thing. George Michaelson - 10:53-11:50 Ah, fluorinert. Yes. It wasn't water, it was a complex PCB that had a vague fluorescent green hue, and it came with a tank with a ball-cock floating it, just like the cistern on top of your toilet. But the IBM mainframe was water. And you know, we retrofitted a cray into a machine room that had had an IBM, and we thought, yippee, we can save all that cost on the engineering. No, no, no, no, no. Because the bend radius on the pipes for water was different to the bend radius on the pipes for fluorinert, and we had to lift the floor. Anyway. Yes, liquid cooling. So you're telling me 60,000 GPS gigabytes of RAM in servers 72 of these GPUs per rack. You can't cool the rack with air. We're moving to liquid cooling. Geoff Huston - 11:50-12:34 Because it's 3,000 times more efficient because of the higher density. I don't actually well I can, but let's talk about it in a second. Normally what she uses is cold plates So you use the same equipment as you always were. And instead of mounting these chips on a large finned thing, you know, fins everywhere, to try and radiate the heat into the the air, what you actually do is you mount these chips beside a plate where inside the plate is a flow of liquid. And it's that liquid that takes the heat from the because it's heat bonded To the GPU, it pulls the heat away from the processing unit and dispatches that hot water to somewhere else to deal with. George Michaelson - 12:34-12:39 Oh, what we might call an externality, someone else's problem. Geoff Huston - 12:39-12:50 In my local town there is a proposal which everyone's getting over from laughing about. where they're going to place data centers beside suburban swimming pools, free hot water. It's insane. George Michaelson - 12:50-12:50 Yeah. Geoff Huston - 12:50-13:29 It's completely insane. But what if I need to get rid of even more heat? Ooh. The more extreme approaches actually say coal plates just won't cut it. You're just not getting enough Tell you what, let's take the entire rack of stuff and pop it in a bath. Not water, because that won't go work, but let's make some non-conductive oil, a dielectric liquid and just simply drop all of that stuff into a bath. Because once we're talking about 140 kilowatts of cooling per rack, cooling. then the only way you're gonna do this is liquid tub cooling. Nothing else is going to work. Wow. George Michaelson - 13:30-13:36 So we've got kilowatts as a unit of electricity measure to make the rack do its job. Geoff Huston - 13:36-13:36 Yes. George Michaelson - 13:37-14:00 And we've got kilowatts as a energy measure, the amount of energy that is thermal that we have to get rid of And there's an efficiency quality here that leads me to suspect if it's off the order, a hundred and fifty kilowatts per rack of thermal cost to be got rid of. I'm actually expending energy to move that heat somewhere else. That doesn't happen for free. Geoff Huston - 14:00-14:32 Right. So now I've got GPUs, I've got high-end switches, I've got laser drivers, all of those things now require cooling. This is one hot thing and there's a lot of power to have to manage and a lot of heat, sorry, a lot of energy I've got to input into it So there's a number of ways we can go there. Where do we get the energy from? You could probably mount a solar panel on the outside, except that you'd need a few hundred acres or more of solar panels just to get even close. to the gigawatt of power you're going to need. Oh, and by the way, George Michaelson - 14:32-14:33 But you said earlier. Geoff Huston - 14:33-14:35 Seven by twenty four. George Michaelson - 14:35-14:35 Yeah. Geoff Huston - 14:35-15:01 So what do you do at night? I don't know. Ignite a thermonuclear explosion and make an artificial sun. Things get weird. very, very quickly. Or you go to your own local module of nuclear power, which of course putting a data center beside a swimming pool makes a whole lot of sense when you also have a small module of nuclear power reactor beside the same swimming pool Very popular with the swimmers, I'm sure. This is weird. George Michaelson - 15:01-15:55 Yeah. Or you do what I believe data centers in the US increasingly have been doing. which is they desperately look to arbitrage sources of power that other people want to sell them, but in order to make sure they have a guarantee of power they install gas turbine local generators, which are very efficient in electrical engineering terms but they have side effects. They're noisy. They're incredibly noisy. You're running a general electric jet engine comparable to the ones that are fitted on a 787 or a triple seven you're running them mounted on the ground to generate electricity. So it's nuclear power, it's someone else's surplus electricity they're trying to sell. but you need to have something else and it's probably a bit of solar, a bit of wind, and some local gas turbines. This is a lot of investment, Geoff. Geoff Huston - 15:55-16:15 It's a massive amount of investment. So large that entire areas in the US can only house one. So Stargate from OpenAI is in Texas. XAI's Colossus is in Tennessee. Meta's Hyperion is in Louisiana. George Michaelson - 16:15-16:26 And each one of these is of the order of scale that you've been discussing that you're going to build for yourself in Canberra. That is how big one of these things are. Geoff Huston - 16:26-16:43 Great project. Yes, exactly. I will take up a large amount of South East Australia when I build mine and no one else can move in there as well because I would have sucked all the available capacity of this system. So yes, that's the level of insanity we are dealing with. It's kind of weird, isn't it? George Michaelson - 16:43-16:50 Now, Geoff, why do you want to make your own AI data center? What were you looking to do here? Geoff Huston - 16:51-17:36 My hesitancy is like many others, I'm kind of grappling with the concept. I really don't know, and neither does anyone else We're building all these weird models and generating synthetic answers based on, well, the magic of neural networking, the idea that if I build a dense enough Internetworked connectivity of the equivalent of silicon neurons, input at one end, output at the other, connect these things in arbitrary ways of complexity and if you put a billion or two together, a bit like a brain, it's gonna work, right? Well, oddly enough, it doesn't do a bad job. It kind of works. Bizarrely. George Michaelson - 17:36-17:54 For some things yes. For some things we're finding that these enormous constructs are helping us create outcomes that we didn't think we'd be able to do. So there is definitely an upside in terms of, I suppose, the creative component. Geoff Huston - 17:54-18:10 But let's just see how much it costs. 'Cause, you know, I'm interested. How much money are we spending on building data centers? Let's take the next five years. Well, the bill appears to be somewhere between three trillion and eight trillion dollars just to build 'em George Michaelson - 18:10-18:12 Three trillion Geoff Huston - 18:13-18:25 Trillion. The GDP of the United States of America is thirty- two trillion. So just to build them is ten percent of their entire GDP. across that period. This is insane and the bill is going to get higher, not lower. George Michaelson - 18:25-19:01 So there are days when ten percent of something is kind of okay. That's all right, I'll pay it. Ten percent of the cost of running my house has to go to insurance. Fine, I'll pay it. It's just the cost of doing business. Ten percent that's GST, the tax you've got to pay it. You plonk 10% of the US GDP on the table as a spend that didn't previously exist, that wasn't part of our normal modeling of the nature of an economy running at 30 trillion. and you say I'm adding ten percent, that is not nothing, Geoff. That has massive, massive consequence to that economy. Massive. Geoff Huston - 19:01-20:36 In previous years, how do we justify this? Well oddly enough, we placed what I would call faith in Moore's Law, because this is a silicon computing problem. that in general you can put more gates in a chip. If you put more gates in a chip, you can do more complexity in a chip. If you then take the same thousand chips, but you have the chips that's twice as capable, that same thousand chips can actually do double the compute. And that's what we've been relying on because the cost per gate kept on coming down year after year after year. So what looked like a scary big number ten years ago is now a tinsy tiny number. What was a supercomputer in the 1960s is now on my wrist as a watch. And instead of costing hundreds of millions of dollars, costs less than a thousand, even so less than a hundred bucks Moore's law made everything cheaper and smaller and interestingly way, way more power efficient. I don't need a mini uh mini nuclear power supply to drive my wristwatch I don't need a lake of ocean to cool my wristwatch. My body can cool it and a little bit of air around it. We relied on Moore's law to take what was at the time. extravagant engineering and by minimizing its silicon footprint we were able to push it down into ever smaller amounts of real estate in the silicon wafer demanding less power while keeping its speed. George Michaelson - 20:36-21:14 You've talked about the effect of Moore's Law before, discussing aspects of things like BGP and routing the scale of the problem, coping with growth in prefixes, getting things done to pass packets on the wire more effectively So the idea that things get better as the technology improves, you've been discussing, but you've also been discussing that we have to accept a reality that Moore's law isn't an absolute law of physics, it's a logistical curve of the nature of technology improvement. And it has potential to slack out, to stop improving at the rate we've been used to. Geoff Huston - 21:14-22:07 Well, you know it's funny, there were two inventions and I would call them inventions because they really were completely unprecedented. And the first was the silicon transistor, Fairchild semiconductor. It came out of Bell Labs in 1947. And it replaced these weird vacuum glass tubes with internal grids and power and heater and so on with a tiny tiny chip of a rare metal bonded to a slightly different rare metal, germanium with different amounts of doping attach conductors to them and they behave like a valve with a whole lot less power and they can switch a whole lot faster than a valve could. That was an invention which, well geez, drove the 20th century. It really did. It was that fundamental and that out of the blue. Silicon just revolutionized our world. George Michaelson - 22:07-22:08 Yeah. Geoff Huston - 22:08-22:21 But that wasn't the only invention. And it actually took another 12 years for someone And I think there are multiples around there in history, but can I build two transistors on one silicon substrate? George Michaelson - 22:21-22:23 Yeah, packaging it up. Geoff Huston - 22:22-22:33 And you think, why didn't you think about that on Thursday? You know, if you didn't think of the transistor on Monday, let's do two, you know. Why didn't that happen straight away? And I I can't answer that George Michaelson - 22:33-23:02 Do you remember buying in the very early days a series of small chips that would typically have like eight or sixteen pins, the seven series, seven, four XX series, and it would literally have on it six NOT gates. Six things that would give you the reverse of what you put in one pin, or six NAND gates, or six NOR gates. You'd buy six of them on one chip the size of your thumb. Oh my god, those were good days. Geoff Huston - 23:02-24:18 Those were good days. But you know, oddly enough, I'll actually class that as another really basic intention. And maybe it's simply because it took about twelve years to figure out we could do that and it was useful to do it. That idea of putting multiple valves or transistors on the same substrate and then manufacturing that collection in one hit. was fundamental. Why do I say that? Well, there's a law called Dykstra's law. I'm not sure if it really has contributed to that computer scientist Ed Ed Edgar Dykstra. But it goes like this. In computing there are only three numbers. None, one, and infinity. Because if we can do it twice, we can do it three times, if we can do it three times, and so on it goes. There's only kind of a unique number or something that isn't unique. There's a lot of them. And interestingly, when we started to say how many gates can you put on a single silicon wafer The answer is, well, there's no limit. Not really. It's all a case of how good your manufacturing process is. So we started to get ten, twenty, a thousand I seem to recall the um 386 processors, the Intel PCs from the late 1980s. If I'm right, three million gates. George Michaelson - 24:18-24:19 Yeah. Geoff Huston - 24:19-24:44 Three million. Could have been more. But it was, you know, getting up there and the feature size was you couldn't see them even with a microscope, we start getting into, you know, millimeter size, then smaller and smaller. But interestingly It's the same technology, it's just more of it. They're planar. You know, it's just a flat surface. It's like a printed circuit board micro edged into a single silicon substrate, right? George Michaelson - 24:44-25:12 And we've been living virtually all of our technology working lives in a railway line that's been running along this path. We know how to do one thing, we'll call it a gate. We know how to do ten things, we'll call it something. We know how to put a thousand of them together, we've made a subsystem, a million of them together, we've made an entire computer on a chip. One million. ten million, a hundred million, a billion, one billion, ten billion, a hundred billion, a trillion. We've been going up this ladder. Geoff Huston - 25:14-26:38 We're up to a trillion. Now, what's the technology behind putting a trillion features on a silicon chip? Because a silicon chip that we're dealing with is not the size of a tennis court. Physically, it's the same size as it always has been, because all the rest of the mechanical processes have been Integration, etc. are geared to that size. So what we do is we reduce the size of the features we're trying to etch. And it's pretty standard etching techniques, but now the feature size is three nanometers. nanometers, three billionths of a meter. That's pretty small. Now how do you etch using photo etching and I use the word photo advisedly, how do you etch features at that size? Well, you can't use light. You have to use extreme ultraviolet. You know, you have to generate light with a wavelength way out there in the ultraviolet. Can't go too far, I suppose, or it becomes radiation rather than light. But you have to go to extraordinary amounts of of sort of expertise To make a light source which has sufficient power that you can etch with. And what we use these days is droplets of Indium tin Vaporized by lasers that generate a consistent powerful light source at 9.64 nanometers, and that will give us etching down to around two nanometers max. That's what we use today. George Michaelson - 26:38-27:04 It's kind of funny that we've simultaneously arrived into a world where mechanical contrivances like carbon arc lamps with an engineer adjusting the edges to bring two carbon electrodes together. No, no, no, that's too crude. Let's make things that don't have any moving parts. We wind up at LEDs. How do you make those things work? Well, you make them by dripping molten tin through a laser beam. That sounds a bit mechanical, Geoff. Geoff Huston - 27:06-28:04 Fifty drops a second. They're discrete drops, and it's not one laser beam, it's three. Each drop gets hit three times. The first one, this sort of tear dray shaped drop, the impulse from the laser, and it's not a continuous, it's an impulse, it's a shot 50 shots a second. The first shot slightly flattens the drop. The second shot turns that slightly flattened drop into a dish. And the third shot, which has a bit more power to it, instantly vaporizes that dish of falling tin into a burst of light. That light is your 9. 6 nanometers. Okay, cool I want to go smaller. Damn it. Well I need a shorter wavelength. Well I can't use Indium tin, so what can I use? I don't know. Does anyone know? No. Keep on melting things and hitting with laser, all you scientists. Hopefully there's something out there that will luminesce when you explode it at a light with an even shorter wavelength. So okay, we don't know. George Michaelson - 28:04-28:23 The idea we're on a rail track of increasing refinement of the feature size down, increasing speed, increasing s cost savings electricity per unit computation is starting to hit roadblocks. We actually don't know how to keep doing that, staying in the flat planar model Geoff Huston - 28:23-29:12 What was fueling this was that we were able to get more precise light when we needed it. We moved from visible light into ultraviolet, then we moved into you know extreme ultraviolet where we are now. And those were quite foundational changes, but the underlying technology etch on the plate was still much the same. But we've got another problem too. How big is a silicon atom? And you've got to go, I'm sorry, small enough, we're fine. But if your tracks only, let's go, one nanometer wide, if your feature size how many silicon atoms can you fit in a one nanometer? And the answer is about twenty. Now, electrons are pesky beasts, and the guys over in the quantum physics lab, damn them, keep on telling us about the duality of ultrasmond, including electrons. That on Mondays, Wednesdays, and Fridays, the bloody things are waves not particles. George Michaelson - 29:12-29:13 Yeah. Geoff Huston - 29:13-29:28 When I'm trying to stop an electron tunneling through a bunch of molecules They behave erratically. If I only have a barrier You'd want a few more than twenty to make your brick wall to keep them where you know they need to be. If I want to reduce the barrier to ten silicon atoms thick, no, it ain't gonna work. You know, quantum effects and everything. George Michaelson - 29:35-29:39 Because on Fridays that electron bounced over the other side and it's quite happy. Geoff Huston - 29:39-30:06 Well it became a waveform and it'll magically appeared at the other side of a supposedly impenetrable barrier of a mere ten atoms. You know, at some point the shrinking of features doesn't work anymore. We are really close to etching into that. So what do we do? Well okay Let's do what we've always done. Let's go sideways. What's sideways for planar? Let's do 3D. Let's build the chips in towers, stacking things up. George Michaelson - 30:06-30:07 Stacking things up. Geoff Huston - 30:07-30:25 That's not easy because what that means is multiple layers of etching. Using a substrate and mechanical etching that I have to now position this wafer layer after layer to an accuracy of fractions of a nanometer. Can we do that? George Michaelson - 30:25-30:38 So the features might be two or three or one nanometers, but to line up a thing that's only one nanometer, you have to be a fraction of a nanometer accurate in positioning things on top of each other. Geoff Huston - 30:38-31:04 That's only if I want two layers. What if I want ten? Then I've got to do extreme accuracy for every layer. just to make sure that the tenth layer kind of coincides with the first and the second and the third, etc. And at this point, let's do a few more little impositions. If you can't make this cheap fabrication machine have a ninety percent yield, ninety percent of the chips you make work out of the gate, you know, work straight away. George Michaelson - 31:04-31:05 It's not economic. Geoff Huston - 31:05-31:06 Give up. Yeah, it's not economic. George Michaelson - 31:07-31:40 Yield has been the bugbear of chip manufacture from the very beginning. When they actually started on this journey of just banging together the two lumps of silicon and germanium. they didn't even know whether or not they'd have a working junction sometimes. And then you wind up in a state where you've got this flat disc of pure silicon. nine nines of purity and for some reason you don't understand half the chips on that disc just don't work It's economically unfeasible. That life track of getting thing up to ninety percent, that can take years. Geoff Huston - 31:40-31:43 Do you remember buying overclocked processors? George Michaelson - 31:43-31:43 Yes. Geoff Huston - 31:43-32:04 The thing had a spec to go, you can put a clock of 300 megahertz 300 million cycles per second into this computer and that's its design parameter and it will work. But over in the chip fabrication laboratory, they were sitting there going, well, what if we turn up the clock to 500 megahertz? Most of the chips wouldn't work, but some did, way beyond the original design spec, but they worked. George Michaelson - 32:04-32:05 Yeah. Geoff Huston - 32:05-32:16 And so the dear folk at the chip manufacturing plant said Let's put them to one side, call them super powerful extra special, call them overclocked processors, and sell them at a premium. Yeah. George Michaelson - 32:16-32:28 Yeah, and there was also the bin where they said these chips we said work at three hundred megahertz. Well these ones don't. They work at a hundred and fifty, but we call them low power chips. And you can run these ones. Geoff Huston - 32:28-32:57 That's right. This this is the reject. Let's get some money for 'em because otherwise we'd have to throw them away. So it kind of points to a process where we're pushing at edges of what we used to do for the last sixty years. And and I suppose the problem is that having done it for sixty years, the amount of interest investment and attention to rapidly different alternatives has all but dissipated. George Michaelson - 32:57-33:05 Yeah. We've driven down a path and walking back out of it invites huge questions. What does that look like? Geoff Huston - 33:05-33:17 What does it look like? What is the other path? Is there another Who's working on other paths? And the answer is, well, no one, dude. This one's been running for 60 years. What's your problem? It just kept on giving until it didn't George Michaelson - 33:17-33:18 Yeah. Geoff Huston - 33:18-33:22 So let's go back to our dear friend AI data centers. George Michaelson - 33:22-33:54 So back up the stack of size from nanometers to a sixteen football field sized factory with no windows, consuming the entire electrical budget for a small city and inside it is 60,000 GPUs in racks associated with gigabytes of memory in computers in racks. And every single one of these things, the memory, the computer chips, the GPUs, is an instance of this problem. How do I make it faster, smaller, cheaper? Geoff Huston - 33:54-34:48 Oh George, you're just telling me I'm crazy wanting one. You're right. And if we'd have had this conversation even twenty years ago, the answer would have been, yeah, you're right, I was joking, because in two years' time. It'll take half the power, half the racks, half of everything, and it will give it the same answer. So why don't we think about this big thing? And while we're thinking about it, the physics and the refinements in silicon chip technology will make all of this really quite affordable, really quite practical, and it won't require some phenomenal amount of investment. But that was a faith in Moore's law. So now if you say, right, it's over, party's gone, we can't do it any better than what we're doing now, you really do need. 60,000 GPUs. You really do need gigawatts of power. And next year, you're going to need the same. And if you want it bigger, You're gonna have to build more of them. George Michaelson - 34:48-35:14 I've kind of got this this left field thought going off here, Geoff. This is a technology show. This is a technology podcast. But you have always been fascinated by economics. It's just something that grinds your gears. You want to think about it and you want to talk about it a lot. And underneath this problem of the technology question, I just can't help feeling there is a huge money question. This is about money. Geoff Huston - 35:14-35:58 It is all about money. So let's go and look at search for a second. I don't pay for search. You don't pay for search. None of us pay per search. Search just works. On the other hand, there's a lot of data centers that have bills. Who pays for search? And the answer is advertising. That was the source that made Google's dreams to organize the world's information and be able to search it in Microseconds, I guess. What turned it into reality was co-opting the world's advertising expenditures and taking a cut as they got spent digitally In the process, we destroyed the world's newspapers. George Michaelson - 35:58-36:07 Yes, I was going to ask about that because this isn't exactly we built it all bigger and everyone magically felt wonderful in this Geoff Huston - 36:07-36:50 There's this phrase, it's the economy stupid. And and the real thing is, it doesn't come from nowhere. We just reappropriate. And what we reappropriated was advertising. So we destroyed newspapers. We've destroyed free-to-wear television. We've taken all their revenues. All of their revenues. And made it into digital advertising. Google and Facebook have divided up the world's advertising budget 50-50 And, you know, that's where the money has gone. That's where it is. Now, search costs Google or similar functions to other folk. A little under a third of a cent. I make it's all proprietary and there's a lot of wild guessing, but it is fractions of a cent. George Michaelson - 36:50-36:52 Per transactional unit per individual. Geoff Huston - 36:53-37:02 Per search. All the world's advertising pays for transactions whose underlying cost is small fractions of a cent. George Michaelson - 37:02-37:04 But there is an awful lot of them. Geoff Huston - 37:03-37:28 Well, that's why we take all the world's advertising. So now let's add AI. Now, will it still cost a third of a cent? To generate one of these AI answers? Well, if you look at the amount of money we're spending building AI data centers, obviously that's not true. Obviously. And the issue is how much more expensive is an AI answer? compared to a simple search answer. And estimates vary. George Michaelson - 37:28-38:10 That's not a simple answer, Geoff, because there's so many input differences and there is at least two radically different models emerging in this world. There's a model that believes very much in the Western sphere of influence, in building the hyperscalar centers you've been talking about towards a certain kind of AI outcome with certain cost. And there's a Chinese version which is saying, well, if you only want it to do a very specific set of things, we can actually pack this down quite a lot and make it smaller. But it only does that thing. And so there's two tensions in this. Putting all of that to one side, there's no way this is costing 0.3 of a cent per question, is there? It's a lot more. Geoff Huston - 38:09-38:35 Even if it's two cents or three cents, even if it's ten times more expensive. But let's go to a dollar, three hundred times more expensive. The answer's somewhere there and in some ways it doesn't matter. It's a much bigger number and advertising can't Pay. And when we go back to, yeah, that that big number, the economy stupid, what are you not going to pay for to pay for AI. George Michaelson - 38:35-38:42 If the money's going to do this job, that money isn't available to go and do other things. Geoff Huston - 38:42-39:03 to do what it's currently doing, a thing. And previously the thing was newspapers and free to air television. But that's really just around search and search infrastructure. I now need ten times the money. You gonna go without food? Probably not. Clean water? Probably not. Power? Well maybe you've given it all to the AI data centers, maybe you will. But you see where I'm heading? George Michaelson - 39:03-39:05 Free heated swimming pool. Geoff Huston - 39:05-39:14 That's the answer. That's the answer. I spend so much on swimming these days that you know turning into a free-heated swimming pool that'd pay for our hour wouldn't it? No, it would not. George Michaelson - 39:14-39:14 No. Geoff Huston - 39:14-39:50 So it's kind of what part of the economy are we willing to chop off and forego to fund this. And ultimately it's not a a big economy question. It's back to consumers. Because what we've actually done is replaced the convenience of packaging up news and information. You might have disagreed with the editorial approach. But newspapers fell, you know, had a role for their sheer convenience and ease of use. It got delivered to you, you spent five minutes scanning it over, you were then an informed person. We haven't got that anymore. You've got to go hunting. George Michaelson - 39:50-39:50 Yeah. Geoff Huston - 39:50-40:04 Go hunting. And it's takes much more effort. The quality is way more variable. And quite frankly, I think most folk would agree it's a meaner, more impoverished world. Without that input, it really is, and that's a cost. George Michaelson - 40:04-40:53 There are other aspects of this cost problem that I think we all see. don't upgrade my computing equipment every year, but I do nonetheless participate in the domestic computing economy. And it used to be that Moore's Law outcome, the one I buy this time, is three times as fast. The one I buy this time it's as fast as it used to be but its price is a lot lower. I've suddenly noticed the things I want to buy have stopped getting cheaper per unit advance in technology. If you want to buy this year's generation of Apple iPhone It's not a cheaper unit. It's twice as expensive. Because the cost of the components, the input stream, the logistics behind making that phone AI capable has gone through the roof. It's incredibly expensive to me now. Geoff Huston - 40:53-41:27 Right, the underlying supply costs they've been diverted, a lot of the supply you know, systems have been diverted into AI, which means consumer goods and other forms are now to pay a premium just to get access into that supply life. So today's phones have the same capability as last year's. Oh, this one folds. Oh great. Oh, this one's colored red. Oh, fantastic. Is it any more powerful? No, not really. Because it's the same underlying technology. And the real answer is this is not a deal that consumers will, you know be hoodwinked into easily. We're not going to replace as frequently. George Michaelson - 41:27-41:28 No. Geoff Huston - 41:28-41:45 That money which was fueling that industry is now being diverted into What, AI? And it's not just that industry, because that's nowhere near, nowhere near the trillions of dollars we're going to need. Nowhere near the 20% of GDP that we're after. George Michaelson - 41:45-41:46 20%. Geoff Huston - 41:46-42:00 20%. This is now big. And the real issue is, at what point and how does this become viable? And it's from an individual consumer's perspective, what are you going to give up? What are you going to forego? George Michaelson - 42:00-43:39 The viability here, kind of, that's a multifaceted statement. If I I'm thinking of making investments in the world using my my income and I'm looking at the return on investment I'm going to get. There are things that I put money into through my superannuation fund with an expectation of growth in its market or stable profits in selling goods and services. And there are other things that I think in terms of, well, it's a bit risky. I might lose some of it, but if it works, I'm going to make a lot of money. AI is a new class of thing that if I'm understanding this right, it now actually is a component of investment that my super fund must invest in because it has become so big it's been listed in a bucket of investment that is judged must be as part of what the institutional funds invest in. And I'm looking at that going, there are some rather weird effects here. You haven't told me that any of these companies are yet making profits in this, but you've told me they're digging giant holes in the ground and they're chucking things into it. And I've noticed Things that I want to get done, like getting a house built or getting an electrician, it costs me a lot more to do that now because those people and those inputs that were available to fund the domestic build economy They're not there. They're being used in the DC, building the DC. So I'm in this weird paradoxical situation. Everything I want to do has got more expensive and less profitable And I'm being told I must invest in the AI. And it feels a bit like it's risky investment, Geoff. It hasn't made any money yet. So why would I do that? Geoff Huston - 43:39-44:32 You see there's another side to this and I actually think, sadly, this is the major motivation, I can replace a whole bunch of human workforce in my company All those layoffs we've been seeing over the last two to three years, all those coders, all those content generators, all those people. I don't have to pay for. I can put AI to the work. So is that 20% of an economy? Well, not not so fast. What are you going to do with those people? Do they become a burden on our social support infrastructure? In which case, zero sum gain. It's somebody else's problem called the taxpayer, corporate tax and everything else. You're not any further ahead. Is this just one of those more cynical views that the job of the Internet and technology has been to make all of the problems somebody else's and all of the rewards mine. George Michaelson - 44:32-44:33 Yes. Geoff Huston - 44:33-45:04 So that you take the bits that are attractive and can generate the revenue and take all of the side effects and all of the negativities And simply throw it back somewhere else, preferably the public purse, but it doesn't really matter, somewhere else. And is AI just a gigantic con game along the same lines? That the real impact is actually not danger to humanity unless you consider absolutely structural no hope of the job in the rest of your lifetime a danger to humanity, then okay, that's what we've got. But that's exactly George Michaelson - 45:04-45:06 I'm I'm kind of there. Geoff Huston - 45:06-45:41 Kind of the that's exactly where we're heading. Because that's the only way. The only way You can justify this mind- boggling cost. The real outcomes of this are at the very worst forms of abuse of our existing social infrastructure. the things that make societies work. That this greed from a few to corner this source of revenue comes at a cost that destroys the very fabric of society they're trying to succour. It's the ultimate parasite that destroys its host. And the real issue is, does the host survive while he enoyed the riches or do we all die together? George Michaelson - 45:41-45:45 Oh, goodness me, what a happy conclusion. Geoff Huston - 45:45-45:49 Isn't economics wonderful? Economics the dismal profession. George Michaelson - 45:49-46:42 I think that there are aspects of this that we really do need to be worried about. When you said a state in America can afford to have one of these in terms of its impact on social infrastructure, electricity, cooling, water supply, technology burdens. The fact is that we understand there isn't only one or two people seeking to make this play. There are hundreds of people seeking to make these plays, but the likely outcome is that a much smaller number of them succeed. And so I'm looking at that number you floated, somewhere between 10 and 20 percent of the national economy is heading into this space and I'm thinking, but if seventy- five percent of those companies are going to crash and burn, you've just said that seventy-five percent of The GDP coming from this space is going to fall in the hole. Geoff Huston - 46:42-46:42 And George Michaelson - 46:43-46:56 that is a large shock to a national economy. If you take two or three or four or five percent of the national economy and say, I'm sorry, we were wrong, that value doesn't exist anymore. That's huge, Geoff. Geoff Huston - 46:56-47:26 It's a classic, isn't it? It's at a scale that societies are going to find that very, very hard to cope with. This is, you know, the mother of all recessions. This is a shock where there's no simple remedy because all of a sudden sort of the productive instruments of a society are being torn asunder. And The ones that fail fail taking a few trillion dollars with them of value. The echoes of empty data centres are just, I don't know, monuments to folly. But they still cost that money to build them. George Michaelson - 47:26-47:27 Yes. Geoff Huston - 47:27-47:28 You know, that money is gone. George Michaelson - 47:28-47:57 When other things fail, the structural assets left behind, if they're real goods, often turn out to have some downstream use But this chip technology, it has a very strong feel there's unlikely to be a significant recovery from walking into this failed DC disassembling the GPUs, disassembling the switches and the fibre, and finding other uses for them. They're not actually going to come to the fore as a good option for someone else, are they? Geoff Huston - 47:56-48:00 Probably not. They're too specialized. They demand liquid cooling. You know, how are you going to run them at home? You know, that kind of stuff is getting pretty esoteric. So this does look more like. an extreme high-risk venture whose benefits accrue only to a very small sector of society whose downsides are universal and equally as inevitable. This is not an option. Once you start appropriating this much money The outcomes are going to happen. It's not an argument. George Michaelson - 48:27-48:29 It's a when, not if. Geoff Huston - 48:29-48:56 It's a when, and the when is frighteningly close. It's not, oh, it's my kids' problem. Good luck. It's it's our problem. It's a here and now problem. And quite frankly, the distraction over bigger, faster, better, you know, shinier colours, bigger data centers, the world will be saved, is kind of distracting us from the real costs of how the hell do we foot that bill? Who foots that bill How does the bill get paid and who's they're going to end up disadvantaged from that? George Michaelson - 48:56-48:57 Yeah. Geoff Huston - 48:57-49:29 And the answer truly is this mind boggling Well it's so expensive. It's such an investment that it's going to displace a lot of us, the people, the folk who actually make this society up. And that, to my mind, is the real depressing side of this entire foray into AI. I, for one, I'm like, it's cool. I can ask a machine things. And it behaves like a person. On the other hand, I could simply ask you, George, and you'd give me an answer that I'd equally trust. So there we go. Are we better off? Well, don't stop doing your job, George, and we're just fine. George Michaelson - 49:29-49:50 Yeah, there's there's a lot to think about in this, Geoff. I think it's something that we're going to have to engage with and understand. I do think there's a when not if component here. And I think our increasing dependency on AI backed solutions is going to turn around and bite us in the bum. That's what I think is gonna happen. Geoff Huston - 49:50-50:07 We're all doomed. I can only, sadly, agree. And for those of you who have stayed with us and haven't managed to walk away in dismal, dismal, abject sense of doom, good luck for persisting and I hope you have some better ideas of how to survive this than I do. George Michaelson - 50:07-50:08 Have you written this one up anywhere, Geoff? Geoff Huston - 50:09-50:10 Too nervous! George Michaelson - 50:10-50:12 It's been fascinating. Thank you Geoff. Geoff Huston - 50:12-50:14 Thanks George. George Michaelson - 50:14-50:54 Geoff did write up his ideas here and you can read about it on the web If you've got a story or research to share here on Ping, why not get in contact by email to ping at apinic. net or via the APNIC social media channels. Also, remember the measurement@apnic.net mailing list on Orbit is there to discuss and share relevant collaborative opportunities, grants and funding opportunities, jobs and gradual placings, or to seek feedback from the community on your own measurement projects. Be sure to check out the video. out the APNIC website for all your resource and community needs. Until next time.