Geoff Huston - 00:00-00:02 This is an amazing invention. You know, it it transforms the entire system. So that's what we do these days. You know, throw all this stuff under the ocean. These repeaters aren't cheap. but they are magic. Generally every you know 500 to a thousand kilometers depends on your system design and what you're trying to achieve here. They just work. The problem is though How do you protect them? All you need is these thin fibre strands, a bit of silica gel, and a copper jacket. The copper carries electrons to power the repeaters. because you've got to you know have enough power there to beam the lasers into the erbium damp fiber sections, etc. You could just cover all that with some polyethylene and drop it over the the side of the boat and it'll be fine. If I'm dropping it down a couple of kilometers of ocean, deep ocean, you're probably just fine, because there's nothing else down there. It's dark and it's a desert, and as long as nothing else moves, it'll be perfectly fine But of course we don't live down there. The cable stations, you know, the things that pump in signal and pump in voltage live up on the surface. And all of us tend to live in coastal areas. Not only enough there's a lot of shipping, there's a lot of fishing. We've got to cohabit this coastal zone with cables. And you can't just lay a bit of polyethylene. underneath say Sydney Harbour and expect it not to get snagged by whatever it is carrying anchors and so on. So when they get close to the coast and come up from the deep, they start coating. This polyethylene with steel wires. They braid it around so it becomes thicker and thicker and thicker And by the time they enter the harbour, it's something the size of an arm or even thicker, really heavy, because it's got armour plated jacketing around it to try and protect it. George Michaelson - 01:57-01:58 You're listening to The Ping, a podcast by APNIC, discussing all things related to measuring the Internet. I'm your host, George Michelson This time, I'm talking to Geoff Houston from APNIC Labs again. In his regular monthly spot on Ping, Geoff and I discussed the worldwide fiber optical network. From its roots in the Victorian era cable systems through to the modern day. They're really remarkable information machines. Using erbium-doped fiber as a special optical pump to recharge the signals when they get too weak hundreds of kilometers along the fiber They have armor sheathing for all the risky passages where ships may pass, and copper to send electric power down to the amplification systems embedded in the fibre. Although we now expect most content to come to us from data centers nearby, the dependence on a worldwide network of fiber systems remains. It's the glue which connects continents and isolated island communities. The fiber systems are far from cheap They have a lifetime of a few decades, set by these embedded electronics. Fibre investment is lumpy. We're coming to the end of the boom and bust cycle. that was driven by the telecommunications sector, investing on behalf of the phone companies of the world, in the time before the dominance of Google and content providers. Those telcos don't have the kind of capital investment muscle they used to, and so the questions of who will build out the next cycle of optical fiber for the world and why they do it arise Geoff, welcome back to Ping. What should we talk about this time? Geoff Huston - 03:41-03:44 Well, thank you, George, and it's good to be here. Today, we might have talked about it I've passed a few times, but I actually want to come back to the topic of submarine cables. George Michaelson - 03:52-03:52 Ooh. Geoff Huston - 03:52-03:54 The things that hold the Internet George Michaelson - 03:54-03:55 together. Soft, strong, and very long. Geoff Huston - 03:57-03:59 Well, we'll see how long in a second. I'd like to head into the area ultimately of where we are today the fractured world we live in, and what can we do about resilience of this system? All of which are fascinating topics, but in doing it just a tiny bit of research and it wasn't a lot. You know, I used Google. I was looking at the early days of cable systems. George Michaelson - 04:19-04:20 How early is early, Geoff? Geoff Huston - 04:21-04:21 Well in this case it's 1858. George Michaelson - 04:23-04:26 Holy cow, that's a long time ago. Geoff Huston - 04:26-04:26 Well well, the early eighteen hundreds had been, I don't know, this remarkable period. It was the Industrial Revolution and sort of converting the world into machines. and harnessing scene. But at the same time there was this revolution in electronics. And out of that of course came the telegraph system. And it was truly miraculous that you know sending a scrap of paper using some horses was unreliable, expensive, and ridiculous. But if you could string up a conductor a coin, you know, copper conductor or seal on a set of poles between two places, you could send the messages instantly. It was brilliant. It transformed everything George Michaelson - 05:08-05:12 There's stuff I remember from reading history and historical novels that there was a window where people were exploiting the information asymmetry. each side of this kind of system. There's kind of two plays here, Geoff. If you're reliant on fast horses and bits of paper then being in a position to surreptitiously open up the leather satchel and look at the paper while the horse rider's having a break at a posting house, well you learn something potentially to your advantage. So there's that And there's the other side of the system. Once you deploy something like a signalling cable that communicates rapidly, you get the information asymmetry that somebody knows something. a hell of a lot quicker than somebody else. And it seems to have been more than just a fictional plot device people were actually using this to make money. Geoff Huston - 06:02-06:03 Well they do today. It's the same fast traders on the stock exchange, but in that case the information by telegram got there a hell of a lot faster than information you know, with a person on the back of a horse. There were these bank robbers who robbed the bank in one town and hopped on a train. And before the telegram. The moment they did that, they were traveling faster than anyone else could communicate. There's just no other way of doing it. So by the time they got to the next station. No one knew that they'd robbed a bank. No one had the descriptions. They were innocent people again. You know, when you travel faster than the speed of information, you kind of acquire some immunity in some sense. But that's not where I wanted to go with this. Where I was going was that that transformation in America and in England and in Europe, they wanted to do it. between these two vast continents. And it wasn't very long before the Overland Telegraph became the project to join America and Europe. George Michaelson - 07:04-07:07 So they had presumably tried shorter distances. I imagine the French and the British bridge the channel and Ireland would have been connected to Britain. So the principle we can send these signals through water if we can make a a medium that is waterproof but conducts electricity internally. That would have been understood. They kind of got there But there is a it's like orders of magnitude jump from crossing the channel to crossing an ocean. That's huge. Geoff Huston - 07:34-07:35 That's right. They've been experimenting. with undersea cables that were able to carry go short distances, UK to Ireland, et cetera, as you say. So they decided the grand experiment. In 1854, they constructed a cable, a single conductor inside insulation from Ireland to Newfoundland, the shortest distance they could do between the two continents. George Michaelson - 08:00-08:04 A single conductor, but back in my days of doing school work You had to connect both ends of a battery up to something to make a circuit. How can you do this with a single connect? Geoff Huston - 08:13-08:14 Think of the ocean as Earth. The other side was the ocean. George Michaelson - 08:17-08:18 Oh, the water. The water is the other half of your sound. That's right. Geoff Huston - 08:22-08:22 So all you needed was a conductor, steel as they they did they used. And of course they did this in one shot. They didn't understand undersea repeaters or amplifiers, and there are no islands right in the middle of the Atlantic. So You needed a fair few volts to push a signal over that distance, but nevertheless, with a certain amount of care and attention, they constructed a cable They were using around 10,000 volts or so to try and sort of push the signal down that distance, and it worked It worked on the sixteenth. George Michaelson - 08:57-08:58 So they might have pushed ten thousand volts in, but they cannot possibly have been getting ten thousand volts and high current out. It must have been a hell of a loss. Geoff Huston - 09:06-09:08 No, you needed you needed to use, or ultimately as it turned out, you needed to use a mirror galvanometer to get the small variations in voltage that you got at the end to actually receive the signal. But nevertheless, on the 16th of August, 1858, they're able to deliver a message. Wonderful. It worked. And over the ensuing days it worked less well. The signal was degrading. You needed to try and basically up the voltage to see if you can recreate the signal And there were two people working on this. One was our friend Edward Orange Wildman Whitehouse. That renowned English surgeon. George Michaelson - 09:52-09:53 Surgeon. Geoff Huston - 09:54-09:55 Surgeon. An electronic dilettante. He was completely unqualified. And he was, I suppose you consider as the chief engineer, and uh he was in a bit of a struggle with uh someone who was actually more skilled and had more experience in things physical and electrical. Thompson. William Thompson. George Michaelson - 10:16-10:18 Lord Thompson of Kelvin. Geoff Huston - 10:18-10:21 Yes, because he went on and did thermodynamics and uh our Kelvin temperature scale, an absolute zero, and so on, is is all attributable to his work in thermodynamics. But He also invented this mirror galvanometer, which was an incredibly sensitive tool that would work on small fractions of volts that came out the other end. But let's get back to this cable, because White House's efforts to try and resurrect the cable that from the second they started using it started to fade away was to up the voltage. Up and up. 10,000, 15,000 through these induction coils to try and rectify these problems. And of course, what ended up happening? was that in the first section of cable, which was built before the latter sections, the conductor wasn't placed in the perfect center of the insulating material. Oops. And as you up the voltage, it basically blew a hole through the insulator. It wasn't a thick enough jacket, it blew a hole. And you know when I said the other half of the circuit was the ocean? George Michaelson - 11:22-11:22 Yeah Geoff Huston - 11:24-11:24 This is commonly known as a short circuit. And that's exactly what happened. Once the hole had been blown, that was it. The entire cable was a mess. Couldn't resurrect it, failed. Service time? Three weeks. George Michaelson - 11:36-11:42 It's a lot of money down the drain in terms of shipping bits of wire across an ocean to get three weeks of communications, isn't it? Geoff Huston - 11:44-11:46 I don't think they were planning for just three weeks, but yes. Whoops. They had an inquiry, they sort of tried to resurrect all the problems and uh quite frankly, White House's uh reputation was trashed. Deservedly so And while he was still a co fellow of the Royal College of Surgeons, his electrical qualifications I think were took a beating. But it didn't stop the effort. And it didn't take long. They tried in 1865. So much cable they had to use two boats, one in each direction, to meet in the middle. But the cable broke, they lost the end. Oops. George Michaelson - 12:16-12:19 Ooh, ooh dear, that's gonna hurt. Geoff Huston - 12:19-12:19 It did. And in eighteen sixty six they tried again and managed to find each other. Isembard Kingdom Brunel that famed English engineer using the Great Eastern was doing the cable laying on one side. And um the cable was set up and it worked this time, thanks largely to the efforts of Thompson And they used galvanometers rather than just massive amounts of voltage. George Michaelson - 12:42-12:45 So a mirror galvanometer, if I remember this right, is a tiny, tiny physical weight of reflective material suspended on something like horsehair sitting between the poles of an electromagnet and as the current flows through the wire that's been wrapped around it it creates magnetic force which makes that mirror twist. And it's the least amount of movement that you can need in order to detect a signal. Geoff Huston - 13:12-13:15 If you shine a light on that mirror, you can detect the movement of the light even with the mirror swings through very small angles. George Michaelson - 13:19-13:22 And absolute milliamperes of current. Geoff Huston - 13:22-13:22 Yes. George Michaelson - 13:22-13:24 So it's like a a contrivance. But It's not very quick as I recall, Geoff. We like to imagine telegraph as people going dit di da da da da di da di-da I think on this thing it was more like five minutes for the thing to swing over and stabilize, and five minutes for it to swing back. Geoff Huston - 13:40-13:42 I think it was bits per fortnight, yes. It it was incredibly slow because of that. No repeaters. George Michaelson - 13:46-13:47 Five minutes per bit compared to sending it across the water for three weeks in a steam packet, you're still ahead, right? Geoff Huston - 13:54-14:00 Well that was the thing, and that's just why no one was giving up on this and, you know, it started the Great Rush. Cables were built and built and built and built. And first big one, Tat 1, undersea cable used by telephony in 1956, which moved from telegraph to full-blown voice circuits. Quite an achievement. And so we we come to today, and today it's impressive. There is some 607 long distance submarine cable systems in today's world. George Michaelson - 14:26-14:27 Six hundred. Six hundred, Geoff. Geoff Huston - 14:30-14:31 They're all fiber. George Michaelson - 14:31-14:32 Holy cow. Geoff Huston - 14:32-14:33 The total length is one point five million kilometres. Million. you know, moon and back kind of stuff. It's a long way. And, you know, they follow, well, there's two sets. There's one set of cables that follow the coastline For example, one that circles Africa, a few miles offshore, but basically winds around the entire continent, landing at the major population centres that are on the coast. There is a similar sort of system that goes down the uh western coast of of South America all the way down to the far south in Chile, and one that goes as far south as Buenos Aires on the eastern side of South America George Michaelson - 15:11-15:17 I can kind of see, you know, first approximation, some really good reasons why you'd do that. The first one is sort of mechanically, short runs, much easier to do. Should you have a problem in a short segment of cable, you don't have a massive question, how far do I have to go to fix it? It's between two endpoints is kind of a smaller problem to deal with, is that? the power problem of how do I keep regenerating signal and managing things? Well, you don't have to make messy contrivances under the water. You bring it up to land, bring it into a machine room, do the work, push it back out again. And thirdly, economics. Well, you've got the option to monetize use of that cable system at all those points along the way. So I could argue seems like a good model both financially and technically. Geoff Huston - 15:59-16:01 It's it's easy and it's hard at the same time. You know, laying cable on land, it's easy to get to, it's easy to repair, it's easy to power. But every local land council, every local thing I remember when I was working in the telco in Australia, some enthusiastic farmer had used his bulldozer to dig a bloody great hole for some dead animals. and lifted out the cable at the same time. George Michaelson - 16:23-16:25 Who put this piece of string in my hole? Geoff Huston - 16:25-16:27 Yeah, don't you even noticed it. Or the bulldozer certainly didn't. Off you go. And land access, right of way, protection and so on. In America they they like stringing them on poles and a lot of people use it as target practice, you know There are real upsides and downsides with doing land. Under the ocean, there is, in theory, ha ha ha, less to go wrong. Certainly, though, once you've laid it Getting it back up again is is a problem, you'd rather not. And of course you can't, you know, instantly get someone down there to look at it You know, fix up a loose solder joint or anything like that, you know, build it well and sort of toss it over the side of the boat and hope that it stays there But does it George Michaelson - 17:07-17:07 these would not be simplistic one carrier, one simple piece contrivances, would they, Geoff? These are going to be slightly more engineered and complex. constructions these days. Geoff Huston - 17:19-17:22 Well, you'd think so, but I think you'd be kinda wrong. It's still a piece of fibre. single mode fibre, about the size of a human hair, maybe a bit thinner, and you might put six or twelve, but not too many, into a collection that you surround with silica gel and put inside a copper sheath. So it's almost like threading some cotton thread down the middle of a garden hose and then jamming some gel down there to keep them in place. George Michaelson - 17:52-17:52 Right. Geoff Huston - 17:52-17:56 Now of course it's fiber, you don't worry too much about you know, exact dead centre. You don't have the problems that they had in the early days of trying to make sure there was equal insulation everywhere, because the silicon shell is not there for electrical insulation. It's actually there to stop the cables abrading against each other. and a braiding against the copper jacks. George Michaelson - 18:12-18:13 So as things move around. Yeah. Geoff Huston - 18:14-18:16 They'll kind of glide past each other. And then there's a piece of copper. Why? Because we've learned that very long runs without amplification photonically is a really bad idea. It doesn't work very well. You need to kind of re- energize the photons from time to time. On land, you would go normally 100-150 kilometers and then whack another fiber amplifier off the line. Sub-C You kind of hope you can go longer. 500 kilometers, a thousand kilometers if you're lucky. Because you they're expensive things, these underwater you know, repeaters. George Michaelson - 18:50-18:51 We're not talking like a Cisco switch where you terminate the fibre in a box under the sea and you're pumping forty- eight volt DC down to a switch that then regenerates And pushes it out again. These are slightly different amplifiers. Geoff Huston - 19:03-19:04 Oh George, you would be talking the nineteen sixties and seventies, and yes they really, really did, and wow. It's amazing it worked at all. At the time, what they used on the old copper conductor systems was indeed a full-blown convert this in, amplify it out, you know, with transistors, whatever, and send it down the next segment. And yes, you needed a lot of DC volts around the jacket of the conductor in order to do this electro, you know, electric amplification. Then they had electric optical systems. where you still had fiber and you had lasers, but you actually converted the incoming photons to electrons, re-amplified it, you know polished up the bits and signal and then energized the next layer. George Michaelson - 19:45-19:49 Cleaned the insides of the zeros and cleaned the outsides of the ones. Geoff Huston - 19:49-19:50 That's the one. They didn't go very fast. I think their speed rate was about 500 kilobits per second. But then came this remarkable trick. And someone found normally. Fiber optic systems are are impregnated with a small amount, I think it's germanium, and this is clear glass. you know, clear glass. Something like three kilometers of this cable transmits light at the same level as a piece of window pane in your house You know, quarter inch of glasses. George Michaelson - 20:18-20:23 So if they're putting this small amount of germanium into it, it would be for optical reasons. It just has the right properties to control the behaviour of dispersion of the light through this thing. Geoff Huston - 20:29-20:31 It's incredibly transparent. But what's some bright spark found as if you have a section of fibre that you impregnate with erbium and you tangentially aim a laser sort of dusting against or brushing against this piece of erbium-doped fibre which bounces inside this chamber of erbium doped fibre. Now why would you do that? Well, a tired and exhausted photon that comes along the cable and hits this erbium-doped cable section gets re-energized. George Michaelson - 21:02-21:03 Whoopee! I got a kick! Geoff Huston - 21:04-21:04 Yes. And comes out the other end at the same frequency, but amplified. George Michaelson - 21:12-21:17 In the old Wheatstone days, Wheatstone Bridge, Telegraph Thompson days They weren't doing frequency modulated signaling. They were doing amplification modification. It was the wave. And what you're saying here is fiber is operating in the frequency domain We're doing things in the phase of the signal and the frequency of the signal. All that happens when you do this trick with the erbium is that you push it harder, so the amplification happens in the wave height domain, but it doesn't alter the frequency. Geoff Huston - 21:45-21:50 You increase the energy of the light, but you don't change its phase or its frequency. And this is then amazing because you can send 20 photons each or 20 wave systems each of different frequencies and feed them into the same segment of erbium. They'll all come out Identical but stronger. George Michaelson - 22:05-22:08 So you don't even have to do one of these per frequency window. You just have all the fibers go past this contrivance. Geoff Huston - 22:14-22:17 Each fibre is separately amplified But all the channels on that single piece of fiber are amplified at the same time. It is brilliant. George Michaelson - 22:23-22:31 And channels here means separate little optical windows of space in the spectrum that could be put down a fiber. There's no physical channeling here. It's the frequency. Red light this bit, blue light that bit, but more finely grey. Geoff Huston - 22:39-22:39 Exactly. So this is an amazing invention. You know, it it transforms the entire system. So that's what we do these days. You know, throw all this stuff under the ocean. These repeaters aren't cheap. but they are magic. Generally every you know 500 to a thousand kilometers depends on your system design and what you're trying to achieve here. They just work. The problem is though, how do you protect them? All you need is these thin fibre strands, bit of silica gel and a copper jacket. The copper carries electrons to power the repeaters because you've got to have enough power there to beam the lasers into the erbium dirt fiber sections, etc. You could just cover all that with some polyethylene and drop it over the side of the boat and it'll be fine. George Michaelson - 23:23-23:23 Mostly. Geoff Huston - 23:23-23:28 If I'm dropping it down a couple of kilometers of ocean, deep ocean, you're probably just fine because there's nothing else down there. It's dark and it's a desert, and as long as nothing else moves, it'll be perfectly fine. But of course we don't live down there. The cable stations, you know, the things that pump in signal and pump in voltage live up on the surface. And all of us tend to live in coastal areas. Not only enough there's a lot of shipping, there's a lot of fishing, we've got to cohabit this coastal zone with cables. And you can't just lay a bit of polyethylene underneath, say, Sydney Harbour and expect it not to get snagged by Whatever it is carrying anchors and so on. So when they get close to the coast and come up from the deep, they start coating This polyethylene with steel wires. They braid it around so it becomes thicker and thicker and thicker And by the time they enter the harbour, it's something the size of an arm or even thicker, really heavy, because it's got armour plated jacketing around it to try and protect it. George Michaelson - 24:27-24:29 We had a ping episode a couple of years back with Nomay Opolinsky, who's from the University of Paris, who had been looking at fiber break rates. around the world and their differences. And there's one part of the world in the Asia- Pacific footprint that's particularly prone to this problem. And it's the fibre approaches to the northern edge of India and Pakistan because it's a shallow sloping sea. It has this tendency to be shallow water which is economically opportunistic for people fishing and for boats trading. Oh, I think I'll anchor for the night, drop the anchor over the side. Whoops, seem to have snagged something. Never mind if I pull hard enough. I'll get my anchor back. They have lots of cable brakes because it's this huge area of shallow water. Geoff Huston - 25:18-25:21 And of course you can just be unlucky, yes. And you know, ships anchors can get you and they will snag the cable. China in 1948 was an interesting area. With the retreat from the Japanese, China descended into basically warlordism with of fighting factions all over the place. And General Chiang Kai-Shek, one of the warlords, found himself hunted out of mainland China And very quickly, taking a whole bunch of cultural relics from the National Museum in Beijing , found himself on the island of what was then called Formosa, now Taiwan. But it wasn't just that island The good general had also maintained control over some other islands. There were three groups actually, the Matsu, Kinmen, previously known as Quemoy, I think, and Penghu Islands. Now, Matsu and Kinmen are within spitting distance of China mainland. It's not over the horizon and far away, you know. Uh with if you had a really, really low tide, you could even possibly walk there. I don't know They're very close. George Michaelson - 26:23-26:26 They'd be the ones that have the giant speaker system, so they can have doof-duff battles, but instead of music, it's politics. Geoff Huston - 26:30-26:31 Oh, you can do very easily. It's not far. So up at Matsu There aren't that many people, about 15,000, I believe, but they're connected by two and soon to be three fiber cables that run from the island of Taiwan up to these islands and interconnect, which is great. Interestingly though, they're remarkably unlucky. Now, it's not a deep ocean, 60-70 meters deep, but in that four years between 2018 and 2022 Those cables between Taiwan and Matsu were cut twenty-seven times. George Michaelson - 27:03-27:04 Twenty-seven. Geoff Huston - 27:04-27:05 It's very unfortunate. George Michaelson - 27:05-27:06 Yeah. Geoff Huston - 27:06-27:08 And even in the last few years A further three outages in the last two years, major outages, things where you have to send the boats out, find the brakes, lift them up, do the repairs. George Michaelson - 27:16-27:16 Yeah. Geoff Huston - 27:16-27:16 And It's kind of why are they so unlucky? George Michaelson - 27:20-27:23 Well, there might be reasons, Geoff. Geoff Huston - 27:23-27:25 There's a lot of fishing going on. Th there's it's it's not just luck sometimes, but there's a certain amount of finger pointing going on And then we look a bit further over to another heavily fibered piece of pretty shallow ocean, otherwise known as the Baltic Sea. And around the Baltic Sea are many countries: Sweden, Finland, Russia, Estonia. Denmark, Germany, Poland, you know, lots of cables. George Michaelson - 27:48-27:49 Lots of gas pipelines. Geoff Huston - 27:49-27:50 Oh, that too. But just in terms of cables, eleven disruptions in the last three years. Aren't lucky? Well, you know. And this leads to an interesting observation of these 1. 5 million kilometers of cable, right, on these 607 cables, they're actually important. How do we protect them? And by important, I really mean important. We don't use satellite systems for mainstream communications on the Internet. It's too slow. It just doesn't work Cables are all we've got. George Michaelson - 28:23-28:34 So we talk about satellite systems and we talk about their bandwidths and speeds, and there obviously has been a significant improvement with the adoption of low Earth orbit systems They have potential to look to an individual consumer or a corporate entity, government research, you name it, as a really high quality, high bandwidth. feed. But I think what you're getting at here is, yes, George, it can do that for the number of people that use it as edge customers. But if you tried to take all the existing fiber traffic and move it over to the satellite segment, it would be swamped. Geoff Huston - 28:59-29:01 Let's just get a bit of physics and reality here, George. Low Earth orbiting. How low? Five hundred kilometers? Okay. So this satellite is basically with a long enough arm you could reach up and touch it five hundred kilometers. How much of the Earth does it see? Well it sees a circular area of Earth with a radius of 90 kilometers or so. Because it's really close and the beam is tightly focused. What does that mean? that satellite system will get you basically across town. It won't go that much further because it's really close. So how do I bridge the Pacific Ocean? Whoops. Those low-earth orbiting systems, a single satellite can't receive from Japan and transmit to Canada from the same spacecraft. To get that degree of vision, the satellite needs to go high, very high, 38,000 kilometers, up into a geostationary orbit With the satellite appears to hang in the same spot in the sky because it rotates at the same speed as the Earth, but then that satellite's a long, long, long, long way away. lousy performance. And then you say, well how much bandwidth can I get out of it? And the answer is not a lot. A few hundred megabits a second. Maybe a gigabit if you push it these days, electronics, but it won't take the world's communications systems and it's only one satellite. You know, you can't repeat what you do in a fiber. Single fiber , two terabits a second. Enormous capacity. And you're trying to say I can take all of that capacity and jam it up into the sky. If I'm using low Earth orbiters, I've got to get the satellites talking to each other and relaying messages. SpaceX use currently I think it's 100 gigabit optics, so 100 gigabit photonics to speak to each other It's not a lot. It hasn't got the capacity. George Michaelson - 30:55-31:01 We're two orders of decimal magnitude off getting close to the signal bandwidths we need. Geoff Huston - 31:01-31:02 Right, so satellites aren't the answer. The real issue is if you snip all the cables from a country, it goes dark. Why does it go dark? We have caches, isn't the entire world served by data centers? Well yes and no. There are things like the DNS, things that actually require immediate and direct connectivity that you can't cache, you can't fake out. And it's that that causes everything to collapse. So they're critical. They're really important. George Michaelson - 31:32-31:33 That even if you had a disconnected nation-state there are enough paths through the DNS that depend on getting outside the nation state to find the anchor information that comes back inside. Things inside are islanded and broken. You could have two things across town, but the naming system, the failure to communicate, means they can't find each other. Geoff Huston - 31:55-32:01 They can't find each other, they can't figure out the translation of a name to a service. They lose it. They just can't participate in the Internet without the DNS. And that's the of those first critical part of what makes the Internet work. So if you cut all those connections, you haven't got that anymore, you go dark, and that's pretty scary. That's catastrophe scenario. So how do we protect it? What do we do about this? Well, we've known for a long time that these cables are really important. And in the same time as it's a defensive problem, it's also an offensive opportunity. Just to illustrate that, war was declared in nineteen fourteen in August. Britain declared warring on Germany, I think, was one of the triggers. There was a bunch of folk getting very enthusiastic. But the fourth of August is interesting because uh the British ship The Alert, oddly enough, sailed out from Dover into the English Channel, and it was carrying grappling hooks. dragged them along the sea-floor and had a little map of where all the German cables were, transatlantic and some of the European cross connects, hooked five German cables, lifted them up to the boat, cut 'em, and dropped 'em back over the air the overboard. Oops. Quite deliberate, what did it do? It forced the Germans to use radio systems to communicate which then became a cryptography problem. Why did America join the war again? Because it was trying to maintain neutrality. But there was a rather incendiary telegram. George Michaelson - 33:26-33:28 Ah, the Zimmerman. Geoff Huston - 33:28-33:29 Called the Zimmerman telegram. And successfully decoded. But the thing is, why wasn't it on the cables? Because the British had snipped them The communication was forced up into radio and the interception systems, and I think it was called Room 40, cracked the code and out came the Zimmerman George Michaelson - 33:46-33:47 telegram. So pretty much for the whole of the modern era 's dependency on semi-instant communication, we've had this exposure of risk. If there's a context of war and strategic need Things like cables are known to be worth attacking. The risk is there. Geoff Huston - 34:03-34:07 Well, the British did it again, September 1939. Good enough in 1914, good enough in 39. And you sit there and look at the Baltic and you look at the China Sea and you go, Well, accident or deliberate? Are we just rehearsing? Because and the other way of looking at this is that the world is strung together by only 600 cables. Only. How much would it take to cut them? And that's not a thought worth pursuing that far because not much. George Michaelson - 34:30-34:35 We probably don't have 600 cable repair boats available, do we? Geoff Huston - 34:35-34:36 Well that's another problem. We will get into this other problem. So, you know, yes, we can try and protect the cable. cover it with sheet um armor plating, you know, steel cables, try and make it resilient, but quite frankly, with a good angle grinder and a big grapnel, they're still susceptible. And the issue is, you know There's also fishing vessels with fishing equipment, and they're big, heavy things. So, yeah, the cables are vulnerable. Vulnerable to accidents and vulnerable to deliberate events at sabotage. So how do you try and get around this? Well, there are a few things you try and do to avoid the problems. Rooting the cable is essential. Try and go deep if you can. But if the two of you are looking at each other across the Baltic, saying go deep is kind of a waste of time. You can't. It's a shallow ocean. And you can't go too deep. Because once you get to a sea-floor that's say five kilometers deep, the Marianas trench, good luck with your grappling iron, it's just too far down there. You're never going to get anything. So you need it kind of Goldilocks deep Indeed it deeper than a continental shelf, but not so deep that you'll never find it again. Okay? Now what? geology. There is this unfortunate problem that the world is not fixed. America hates Africa. They're drifting apart at the rate of a few centimeters a year. The mid-Atlantic trench is the result. There's a rift is the result of all this. Plates move and things move. And we have earthquakes and geologic activity because of this movement. And laying cables across active geology can be a problem. In I think it was 2007, 35 cables got snapped when there was a submarine landslide as a result of an earthquake just south of the island of Taiwan. Why was that important? Because how to get from the inner sea, the South China Sea, across into the Pacific Ocean. There are only a few gateways you can use. And that gateway of the Luzon Strait was used by many cables. And when there was submarine geological activity, they all went. at once. George Michaelson - 36:43-36:50 Now, I believe there's a bit of a story hidden in that, Geoff, because one of the aspects of designing a cable system is that you mentally, at kind of the physical layer one you think of it in terms of a loop. It's not just one piece of wire. You generally have a loop, an alternate that you think about as the other half of this system that you can do some things with. And I believe for the loose on straight the alternate path that was being offered by many cable providers went down the same physical point. It was two cables running down the same exposure of geological risk. Geoff Huston - 37:22-37:23 That is true. Can I park that thought for a second and get back to it? George Michaelson - 37:26-37:26 Yes. Geoff Huston - 37:26-37:32 Because I just want to get rid of the issue of unavoidable geography and geology first. And then we'll talk about loops and rings. The Luzon Strait 's just one example. How do you get from Asia to Europe? Well you can go around the bottom of Africa. That's incredibly long, expensive, because distance costs, and it's slow. Or you can do what shipping tries to do. You can go through the Sewers Canal. How do you get to the Suez Canal? Well, the first thing you've got to do is go through the Bab El Mendeb Strait, and indeed 16 cables pass through them. The Bab El Mendeb Strait is really, really narrow, and they seem to be having a good time in weapons and throwing bits of weaponry at each other in various ways and forms. Would you put a cable through there today You'd probably try not to. It's not a good idea. George Michaelson - 38:17-38:20 I don't think I'd want to be working on the cable ship laying the cable through there right now, Geoff. I wouldn't go there. Geoff Huston - 38:25-38:28 What if you're the one remaining cable repair ship operating in the Red Sea? So the so the um Persian Gulf, isn't it? George Michaelson - 38:33-38:34 Yes. Yes. The Straits of Hormuz in that area. Geoff Huston - 38:37-38:37 Well, just up up the Gulf from the Straits of Hormuz, there's only one repair ship and they can't send any more because it's a little bit hot. And even when you get to the top of that straight, the top of that golf into Q8, where do you go? Your choices are limited. But some cable operators have been done a deal and actually trying to do a corridor up through Iraq into Turkey, and all of this is to avoid the Babel Mandeb Strait and the Suez Canal. So geography is a problem sometimes, and try as you might, it's really hard to avoid it. Singapore, much more politically stable these days, and that's fine. There are 35 cable systems that come into Singapore as junction points. That, the Singapore Strait, is incredibly shallow And it's also one of the densest areas of shipping. And what happens with ships when they come to harbour? They throw their anchor overboard. Again, it's a busy area with a huge amount of shipping activity, lots of potential for disastrous situations. So none of this is good. None of it is good. So we avoid land routes because they're expensive, tiresome, difficult. We try and go into the sea because it's almost quieter in some ways and once you do the original design. Hopefully you throw it over the boat and it all just works. But you're not immune from these kinds of problems. So what did we used to do in the phone company system? Because what you didn't have was dynamic routing What you had was circuits. And you couldn't, if you will, detect a break in the conducting systems and have the circuits magically reconfigure. Or could you? You see the phone system came up with reads. George Michaelson - 40:21-40:23 Alternate paths. Geoff Huston - 40:23-40:26 But they were dynamically maintained alternate paths. So if you think of a big circle, and a bunch of stations connected to it, the traffic might want to go clockwise. So when A wants to talk to D, it goes clockwise around the circle, goes past B and past C and gets to D. That's fine, and when D wants to talk to A, it just goes back the same way, this time anti-clockwise, D to C to B to A. This is good Let's say you break the cable between B and C. Whoops, A simply goes the other way. It goes the other direction around the ring and gets to D. And you can make that switch really fast At one point Sonic SDH rings were doing switching in 50 milliseconds. You could keep the virtual circuit intact and automatically heal around the brake. And we started doing submarine cables that way, that we built these things as gigantic ring systems that automatically detect single point breakage and then just healed. But George Michaelson - 41:24-41:24 But there's always a but. Geoff Huston - 41:26-41:27 At the time we were used to paying enormous sums of money to make international phone calls. And it wasn't, you know, a dollar an hour. It wasn't even a dollar a minute. They were expensive and what you were paying for was this remarkable resilience that half of the cable systems were idle all of the time. All of the time. The maximum load a ring could take was half of its theoretic capacity because you needed to have the ability to recover from a break seamlessly and immediately That's fine. George Michaelson - 42:01-42:03 Seems like an opportunity to me. I think there's someone out there who's going to take the risk. If I sell 'em that slack half, I bet you I can find a sucker to sell it to Well Geoff Huston - 42:13-42:15 I was that sucker, I was busy building the Internet and I had routers, and I wasn't trying to maintain circuit state. I wasn't Packets didn't need it. So I was quite happy to buy interruptible capacity. Even buy on the backup systems As long as I bought in enough cables simultaneously, the routers would maintain coherency of my overlay network. And I could tolerate some degree of outage in any individual cable system. We'd shifted the onus of repair and protection up a level. The cable didn't have to be there all of the time. George Michaelson - 42:49-42:50 Yeah. Geoff Huston - 42:50-42:52 As long as there was enough diversity of cables and I had bought positions on all of them My routers could do the necessary monitoring and recovery, and the end applications wouldn't notice because packets are packets. George Michaelson - 43:03-43:05 That's a story you could bring into the modern era. If you want to build some resiliency into this system You wouldn't consider it as only six hundred cables. You'd find a way to say it's got to be at least double or triple that number, and you'd say no single cable should be used to provide service for any one customer. you'd spread the load over the whole lot and make it more integrally resilient. Geoff Huston - 43:26-43:30 But think about it from the point of view of the customer rather than the cable operator. As a customer I buy positions on three different cables. in the full knowledge that any of those cables will break. You know, I'm just betting or hoping that all three won't break at the same time So let's not talk about the lose on straight anymore, because that was a terrible disaster. George Michaelson - 43:48-43:50 Because they did. Geoff Huston - 43:50-43:50 They did. But I'm betting that that won't happen. And it's then my responsibility to patch up. What's the problem? The problem is I might not have enough capacity when I'm using this old mutual backup systems. I might suffer a performance problem. But I don't suffer a connectivity problem. The connection's still there, it's just not in much capacity. George Michaelson - 44:10-44:11 You do also acquire a complexity problem. You've got to do more work or you've got to pay agencies on your behalf. to intermediate some kind of an outcome here. Geoff Huston - 44:21-44:24 I've got to do more work, but I'm not paying for 50% idle. It's cheaper. And the Internet is, above all else, cheap. George Michaelson - 44:29-44:30 Yeah. Geoff Huston - 44:30-44:31 That's all we are. George Michaelson - 44:31-44:33 We should have that on a t-shirt. The Internet run by cheap-skates. Geoff Huston - 44:35-44:38 Run by cheap-skates, and that's all it is. And you know, it has worked. Comms are really cheap these days. They're just not as reliable as they used to be, but we're patching it up at the next level up. Okay, well. Fair enough, you say, will we keep on doing this? Will we keep on doing these systems and building them that way? Ooh, let's look at this again I said one and a half million kilometers of fiber. Yeah? George Michaelson - 45:01-45:02 Yep. Geoff Huston - 45:02-45:03 How long does fiber last? The good news is more than three weeks. This is good. Each cable lasts more than three weeks. George Michaelson - 45:10-45:14 I'm reminded of a joke amongst archaeologists. There's archaeologists in three different economies, we won't name them, and they're digging trenches. and the first one finds strands of copper and proudly announces that he knows his primitive culture invented Ethernet, and the second one finds strands of glass and proudly announces that his culture invented fiber optics And the third one finds nothing and says that's proof they invented wireless. But the thing here is, you do find glass in archaeological digs. Glass mechanically, physically is a construct, lasts in the system a very long time. But not unblemished. Glass is partially soluble, it's an interesting material, it degrades over time, it can absorb water, it can change its properties, its optical transmissive properties over time. So I kind of want to say glass lasts for a million years. Geoff Huston - 46:08-46:09 It's not the glass, George. George Michaelson - 46:09-46:10 It's not the glass. Geoff Huston - 46:10-46:11 It's not the glass. It's the repeaters. George Michaelson - 46:12-46:12 That erbium trick, do they run out of 'erb? Geoff Huston - 46:15-46:17 The huge amount of micro-electronics power in those systems to actually make the cable work over incredibly long runs, deep under the ocean, with beryllium copper and every last trick in the book, means they have a finite lifetime. George Michaelson - 46:31-46:32 Right. Geoff Huston - 46:32-46:34 And so the bookkeepers come along and say, I will deem you to have 25 years of active life. And at the end of those 25 years of active life, we will destroy this asset. It has outlived itself. We believe that by then there will be other assets under the ocean that do it for less, do it better, do it cheaper. So the declared lifetime is 25 years George Michaelson - 46:55-46:59 But it wasn't the glass, it was the associated infrastructure. Geoff Huston - 46:59-47:00 Everything else. George Michaelson - 47:00-47:00 Wow. Geoff Huston - 47:00-47:01 So we built a whole bunch of cable 845,000 kilometres of currently active cable around the turn of the century in the year 2000. George Michaelson - 47:11-47:15 Oh no, in the same narrow window in time. Geoff Huston - 47:15-47:16 Oh yes. The great Internet boom. And so George Michaelson - 47:18-47:19 by twenty Geoff Huston - 47:19-47:20 thirty, four years from now, George Michaelson - 47:21-47:22 they're going to hit their peak ages and acid. Geoff Huston - 47:23-47:27 More than half of the global inventory of undersea cables are due for decommissioning. They're already hauling up TAT 8 and TAT14 to recover the bits of copper because it's good copper and it's worth it. CME W3, dead, gone. It's a whole bunch of these Japan US, China, US, they're gone. George Michaelson - 47:45-47:48 If I flick back to the last episode of Ping we recorded, one of the problems that emerges with this hyperscalar data center is the economic impact that all the capital in the world is being hoovered up into these other projects because people are looking for speculative gain. And you're telling me that a bubble of asset investment made in the early 2000s is about to hit its wall of lifetime and demands capital investment. they Geoff Huston - 48:13-48:14 're going to be turned off. I'm not saying demands capital investment, but half of our inventory, a little over half, is about to go. But it's worse than that It's worse than that. Because when these cables break, shipping activity, anchors, you know, geology, when they break, and they do break, you know, nothing 's perfect, we have to send out the repair ship. Ding ding ding repair ship. How many repair ships do we have in the world? George Michaelson - 48:42-48:43 Well not 600. That was my point earlier. Geoff Huston - 48:44-48:46 Yeah, we only have 60. George Michaelson - 48:46-48:46 Sixty. Geoff Huston - 48:46-48:47 Okay. Well, only sixty, this is fine. But you'd like it because they're valuable assets. They cost a billion or so. You know, you'd like to get out there soon. So you'd like to have a bunch of them nearby. How old are they? Ooh, that's not a question you really want to know, because ships don't last forever. George Michaelson - 49:04-49:06 They don't last as long as the cables. Geoff Huston - 49:06-49:11 More than half of that fleet will have aged out by 2040. In other words, within fifteen years or so, we will not have boats available to repair outages that will occur. Ooh, this is starting to get a bit painful. George Michaelson - 49:21-49:22 This is money. We need money. Geoff Huston - 49:23-49:25 So now we've got this problem. that we need to build new cables. This is good. And we need to build new vessels. This is good. Who's going to pay for them? Now, we have destroyed the old telco economy. There hasn't been a telco consortium cable since 2000. That last boom of building undersea cables was the last great hurrah. Who now builds and operates cables? George Michaelson - 49:50-49:51 Not telcos. Geoff Huston - 49:51-49:53 Or the hyper-scalers. George Michaelson - 49:53-49:54 Not telcos. Geoff Huston - 49:54-49:55 Well not telcos. And you go, well, hyper-scalers. And the real question is, why are you using a plural Why are you saying hyper-scalers? Who is about to own more cable than everyone else combined? was Google. George Michaelson - 50:10-50:10 The only one investing in cable is Google. Geoff Huston - 50:13-50:16 At that level, at that phenomenal level. Now we sit there and go, well, whose problem is it? That the island of Australia ends up being connected to everywhere else by an American company, answerable to American investors. And then you start sort of raising a few eyebrows around sovereign risk. about whose priority is it if any of them go snap? And Google has a commercial problem, but Australia has a much bigger problem than merely commerce. It's almost a digital survival problem. George Michaelson - 50:45-50:47 This problem has been latent in the idea of cables between two nation states from the very beginning, Geoff. I mean the legal jurisdictional qualities of who owns the asset, who has the incurred risk in the asset, who owns the cost of using the asset, the information, the responsibilities. It's always been there. But it seems like you're saying with the shift over to hyper-scalers, the component of inclusion and involvement by Australian companies, the Australian government, the Signals Directorate has kind of dropped down a bit We haven't kept our eyes on the ball here. Geoff Huston - 51:20-51:23 Well, let's take that theme and amplify it. Any society to function needs to be able to communicate and communicate well. You could argue that one of the biggest problems for the Inca civilization was the inability to actually have written written information. Knots in string is all very quaint, but it's hardly uh the way to build empires. So what you need is open, accessible, and efficient communications. for your society. Not anyone else's, for your society. And in the early days when we started to toy around with telegraphy. with telephony and so on, we made it a state function, a function of our state. Why? The costs were high, so pulling the asset seemed to work. But equally the ethos of a public system matched its requirement in a society that thrived. And that worked for a while through a lot of the 20th century until. George Michaelson - 52:15-52:16 For most economies this was the utility function. It's the public utility function. Geoff Huston - 52:21-52:22 It's the same as letters. It was absorbed a lot by the Postmaster Department in many, many countries as the model Right? But there was an aging of technology and a need to reinvest. And it became obvious with transistors and electronics in the late 70s, early 80s, that the telephone system needed reinvention. And at that point, the public purse said no, in a very adroit move that I'm sure the whiz kids of the day thought was the best thought they'd ever had since Wheat Beaks or something. They decided to make this a burden of the private sector. Let's deregulate the telephone companies and let private money build the next generation of public communications. George Michaelson - 53:05-53:10 There was a wave of belief that it was time to privatize. all of the utility functions. So we see the electricity distribution network, the gas networks, the telecommunications, water, sewerage. all of them being moved out of classic public ownership into private asset models. Geoff Huston - 53:24-53:26 The discipline of competition will keep them honest. All we need is a residual market monitoring function to enforce you know, fair competition and fair rules amongst private players and the public will be served efficiently. George Michaelson - 53:37-53:41 Excuse me while I laugh holding up my most recent bill Geoff Huston - 53:41-53:48 Well, excuse me, excuse me while I talk about the single hyper scaler left in the submarine cable sector. George Michaelson - 53:48-53:49 Right. Geoff Huston - 53:49-53:49 And wonder what form of competition are we talking about? Is it Google Department A competing with Google Department B? I think not. George Michaelson - 53:57-53:57 Yeah. Geoff Huston - 53:57-54:01 That what's actually happened, particularly in the international realm. where normal mechanisms of control never existed very well, is that the whole system has come completely distorted. And this has some remarkably alarming effects because now we have a communication system which is actually not just capture a bull. but captured. And we have an undersea cable system which is incredibly vulnerable to deliberate acts of sabotage when things get nasty. And things are nasty today. It's not a benign world. So when the EU makes a vote that we will spend a few hundred thousand euros in, you know, shoring up repair vessels and submarine cable systems. Does that make sense if all they're doing is sending money to Google? That makes sense because they have no national sovereign infrastructure directly under their control anyway. So this is a really, really difficult topic. But all of a sudden what we have is an infrastructure we all rely on incredibly that has now been, if you will, maneuvered and altered through economics and politics into a small number of clique players possibly only one if you think about it, which are answerable to George Michaelson - 55:16-55:17 Yeah, who? Geoff Huston - 55:17-55:18 Well, I don't know, George. And when we talk about resilience, Is it resilience against random acts of sabotage or resilience against capture by a single monopoly operator of the global communications endeavor? Which is worse. George Michaelson - 55:33-55:36 That wasn't on my dance card for this decade, Geoff. I wasn't planning on having to worry about that. Geoff Huston - 55:39-55:42 Well, I hope you can sleep tonight, because I'm sure I can't either. George Michaelson - 55:42-55:42 Ha! That's been absolutely fascinating, Geoff. Can we read about this on the blog? Geoff Huston - 55:47-55:49 I have indeed written a blog on this on Potaroo. net that goes through some of these salient points in a bit more detail. I've tried to make it light reading, but quite frankly, it's very hard to hide jokes inside this rather depressing situation. So yes, it's all there. George Michaelson - 56:01-56:06 We haven't arrived at a place that I think either of us would have expected two or three decades ago. But it's arguable the trends were there. We just didn't really appreciate how strongly they were happening. Geoff Huston - 56:12-56:13 Well the big issue about the Internet that I think all of us fail to appreciate at the time is the question, what if everyone did it? And we were kind of threatening it, going, well we'll take over telephony. The Internet will change the world. It will liberate information. It will influence a new digital societal age, an age of equality of access to information and so on. And what we've done is the very, very worst thing we could have possibly done You could use the catch-phrase digital slavery and she wouldn't be far off. That we've managed to create a world that's totally unanticipated by everybody. And now the real question is I don't think we want to be here. Where are the exit points? How can we moderate this world? How can we alter this situation without bringing everything down on our poor heads? George Michaelson - 56:58-56:59 Yeah. Geoff Huston - 56:59-56:59 Yes. Tough question. George Michaelson - 57:00-57:01 Big question. Geoff Huston - 57:01-57:01 And uh Dear listener, good luck. George Michaelson - 57:03-57:04 Thanks, Geoff. Geoff Huston - 57:04-57:05 Thanks, George. George Michaelson - 57:06-57:12 If you've got a story or research to share here on Ping, why not get in contact by email to ping@apnic.net or via the APNIC social media. media channels. Also, remember the measurement@apnic.net mailing list on Orbit is there to discuss and share relevant collaborative opportunities Grants and funding opportunities, jobs and graduate placings, or to seek feedback from the community on your own measurement projects. Be sure to check out the APNIC website for all your resource and community needs. Until next time.