Marc Blanchet 0:00 IP doesn't have any notion of time, right? [George: No]. So a pack an IP packet and even with UDP payload, both don't have any time in it, right? So an IP packet could live for a year, right? But IP and UDP are unreliable. So the core question there's essentially three things you need to care about for Deep Space, IP networking and the core one is reliable transport. How do you make sure that transport behave properly? With those considerations of delays? [George: And the other two], the other two so for transition period and transition in space might be, we'll see, but sometimes we will have orbiters or satellites around Moon and Mars that would not necessarily cover everywhere, right? There will not be a constellation like Starlink on the earth. [George: So partial coverage, partial visibility], right? Therefore, you know, use communicate, for example, from Earth to the satellite, but the satellite cannot reach the asset on the moon surface because the asset is on the other side, right, for example, [George: yeah], so you sometimes will be able to reach the destination, sometimes not. Therefore, you'll see that, what do you do? And if the satellite is actually an IP forward or normal IP forwarder, when the destination is unreachable, you just drop the packet Right, [George: right]. That's the way we do it. George Michaelson 1:41 You're listening to PING, a podcast by APNIC discussing all things related to measuring the Internet. I'm your host, George Michaelson, this time on PING, I'm talking to Marc Blanchet from Viagenie, which is located in Quebec, Canada. Mark has over 30 years experience in networking and protocol development in the IETF, he co founded the IPV6 forum and has RFCs back to 2003 in internationalization, IPV6 and many aspects of the protocol space. Most recently, he's been exploring the application of Internet protocols to space communications in the TIPTOP IETF Working Group. Tip Top stands for taking IP to other planets, and it looks at taking the IP stack to the other celestial bodies, such as the moon, the asteroid belt and other planets. I caught up with Mark at the apricot meeting held in Jakarta. He presented at the opening plenary on the behaviors of the IP stack and in particular the QUIC protocol in very long delay channels that you typically find in deep space, sometimes these communications are mediated by intermediate routing services provided by spacecraft. Mark has been looking at which protocols above the IP level make sense to use in space and how we can tune protocol behaviors around window size and re transmission using a novel testbed which mimics the delay and loss behaviors to the moon and beyond. Mark, welcome to PING. Thank you for agreeing to talk to us today. Marc Blanchet 3:16 Happy. George Michaelson 3:17 Can you just tell people a little bit about yourself? Marc Blanchet 3:20 Started working on Internet protocols while I was a student at the University. This was in Quebec. Yeah, Quebec City and the department started to buy Sun workstations were French speaking, so Unix at that time was not working well with accents, so I started to see if we could use French by email. Then got starting to look where you know how an SMTP could work, and we essentially use it because it was passing right away 8 bit, therefore was able to pass ISO, 8859.1 characters, which means that French was for it. [George: Yeah], that was when I started being exposed to Linux. George Michaelson 4:14 And so this is before the MIME encodings [Marc: exactly]. So you were sneaking eight bit unclean payload into smtp mail, wonderful stuff. So we're here. Ping is for measurement, measurement of the Internet, measurement that leverages the Internet. And you have spent recent years working in a particular niche of Internet technology, the idea of the application of Internet to long delay networking, particularly out of space networking, and you've just presented here at the apricot meeting on aspects of applying modern protocols into these incredibly long delay context which raise. Is a really interesting question. Have you actually got someone doing experiments up there in live orbiting satellites around other planets, or is this something that we're having to do simulations and modeling on on the ground? How are you actually exploring the space? Marc Blanchet 5:17 Good question. I sent my daughter to Mars, and she helped me experiment that George Michaelson 5:22 I thought that was Elon in a car! Marc Blanchet 5:27 No, actually, space is very difficult, right? So, you know, as you could see nowadays, even if we landed on the moon 50 years ago, we're still have difficulty to actually land again. At the moment, George Michaelson 5:42 The most recent one that I think I saw, they made a remarkably successful semi soft landing, and then the device immediately fell over. So the opportunities here, windows of opportunity for you to actually get IP active code into space, into, say, a cis lunar orbit, or even further in space. They are very limited, aren't they? Marc Blanchet 6:02 Yeah, well, actually that one which was intuitive machines lander, and one of the payload was actually the Nokia 4g base station, because he wanted to test 3g 4g on the moon surface. George Michaelson 6:18 So there was a deliberate experiment of launching 3g 4g class band, classic and radio system, standard, off the shelf equipment, maybe a little space hardened, and there would be more than one active device. Marc Blanchet 6:32 Yes. For four, four nodes, yeah. George Michaelson 6:37 So the layer one, layer two part is kind of designed by three GPP class specs Marc Blanchet 6:44 Yeah, there was no things for space apart from ARIN. George Michaelson 6:49 So who did the address in here? Was there, like, a DHCP server? Marc Blanchet 6:54 Yeah, that that? I don't know. I haven't, but I don't think it mattered by that much. But so spaces is hard right? So you you want to the same time, assimilate a lot, as much as you can, right? [George: Yeah] down here, even if you cannot fully replicate the environment. And then, as many have done, which is, do this and step wise, right something, and you go further. George Michaelson 7:24 So the classic problem here is that space is huge, and we're used to the idea of geosynchronous orbit satellites running IP technology, and that's a delay component, which, as users, our personal experience is, well, it's a bad delay for voice, but if you're actually doing directed IP like DirecTV, even geosynchronous orbit is really quite remarkably tolerable in terms of the kind of packet rates you can achieve through it. But that's close to Earth. Moon. Moon is a significantly higher delay, Marc Blanchet 8:00 couple of seconds, one way delay, George Michaelson 8:02 one way delay of a couple of seconds. So that's a very long time to have things in flight and then discover on arrival you've got corruption. And it feels, I mean, this is the naive view, but it feels like those increasing distances are a massive problem for a protocol that has to have reliable packet delivery. Marc Blanchet 8:23 And the last three words are the important ones, right? Because IP doesn't have any notion of time, right? [George: No]. So a pack an IP packet, and even with UDP payload, both don't have any time in it, right? So an IP packet could live for a year, right? But IP and UDP are unreliable. So with the core question, there's essentially three things you need to care about for Deep Space, IP networking, and the core one is reliable transport. How do you make sure that transport behave properly with those considerations of delays George Michaelson 9:03 George: and the other two? Marc Blanchet 9:04 the other two so for transition period and transition in space might be we'll see, but sometimes we will have orbiters or satellites around Moon and Mars that would not necessarily cover everywhere, right. There will not be a constellation like Starlink on the earth. George Michaelson 9:23 George: So partial coverage, partial visibility. Marc Blanchet 9:26 right? Therefore, you know, use communicate, for example, from Earth to the sunlight, but the satellite cannot reach the asset on the moon surface because the asset is on the other side, right? For example, yeah, so you sometimes we'll be able to reach the des nation, sometimes not. Therefore you'll see that what do you do? And if the satellite is actually an IP forward or normal IP forwarder, when the destination is unreachable, you just drop the packet Right, right? That's the way we do it. George Michaelson 9:58 Classic IP behavior. At this point is wanting to say, can't complete. But this is a highly predictable situation. It's not has gone and will never be seen again. It's has gone behind orbit is coming back in an hour. Marc Blanchet 10:12 Yeah. So the first thing you need to do in deep space for the IP stack to succeed is for forwarders that are seeing intermittents, they need to store packets temporarily, temporarily, meaning not nano seconds, but George Michaelson 10:29 Right. So we used minutes. We would be used to the idea that in a congested network with routers and switches, there's a certain amount of buffering, and there is aspects we've discussed on PING before how the behavior of networks ultimately is bound in the size of the buffer and the time window to fill a buffer, but you're talking about a different kind of buffering constraint, and one that isn't measured in quite the same way, because classically, you size your buffer to equal the length of time packets live in Flight in a fiber or in a length of copper, but we're now talking about packets that potentially are in flight for 30 seconds or 30 minutes or 30 hours, and so bringing time to the table seems to change the nature of the equation. Is it really the same? Marc Blanchet 11:19 So back to sizing the buffers. At the same time, we found, more recently, because of Dave Taht, Jim Gettys the buffer block problem, right? You have a large, too large buffer. George Michaelson 11:33 Yeah, it's counterproductive, Marc Blanchet 11:35 right? So you need to kind of queue and do queue management and carry out the old packets and stuff. So we're what I'm proposing is, I said that to Jim at some point, is buffer bloat on steroids. So the first thing you need to do is to store packets right? Second is, if you require and I make the assumption that you do require reliable transport end to end, because George Michaelson 12:09 at the end of this process, Marc Blanchet 12:10 right? Because, you know, typically, we're sending multi 100 millions of dollars of a spacecraft into space, and essentially, what it brings you is data back, right? If you don't provide reliability, it doesn't make sense to me. But [George: yeah], they might be, you know, cases where you don't need reliability, therefore you use UDP, so you need reliability, but then the typical transport has.. so our transport has, does the following, which is handles, packet loss, reordering, duplication, flow control and congestion control, [George: yeah]. So all those things are based on some counters, some timers, some counters, George Michaelson 12:58 yeah. And active flow back from the end point to say ACK right, to adjust window size, to request re crap, transmission, all those things. All of those things are predicated on a complete cycle. You to Mars, Mars to you. Marc Blanchet 13:12 yes, and but if you think about it, really, there's a bit of so essentially, you know, in a nutshell, you need to adjust timers come to the expected RTT of your connections, right? George Michaelson 13:27 Yeah. So this is the part where IP itself has no concept of time, but the higher protocol that provides a reliable basis, unfortunately, necessarily brought time to the table. You have said, however, if you are in a position to tolerate a certain unreliability because of a one way flow, UDP is going to be okay, as long as we have some mechanism we talk about later, maybe to deal with loss and corruption of data. But classic behavior that we're seeing these days tends to be TCP bound behavior. So it's got this end to end, two way dialog going on. And you're saying, Let's adjust timer let's make timers bigger here, yeah, Marc Blanchet 14:08 and be efficient. So and I'll talk about the third thing, and then come back the third thing. So one is store packets when you can't forward. Second, we adjusts the timers or create a profile of the transport for the your use case, which is longer delays. And third is, so when I say I talk about Internet Protocol, people say, you know, my browser won't work in space. Yeah. So you need applications that are tailored for that environment. George Michaelson 14:45 The third thing is applications, Marc Blanchet 14:46 right? So applications, what that means, essentially, is you need to have a kind of a synchronous design, right? You send a request, and then the thread goes sleeping until you get back the response. Right, right. And then the second is, if you have, and that's typical, if you have in your application, timeouts will just then related to the RTT you think you will be happening. George Michaelson 15:12 Something modern users might not appreciate is that long, long ago when Mark and I had more hair and were younger, machines weren't necessarily reliable, and it was actually possible for a computer to be turned off in a hard sense during IP transmission, but it's also possible, if you know what you're doing, to turn that machine back on, set parameters for a TCP session and say, Here's your window size, here's your packet count number, here's your current state in the state machine, and just carry on sending packets. And you haven't necessarily broken that TCP session. So we already have, to some extent, [Marc it's a state] it's just state. If you've kept state, things could work so without sending your daughter in orbit out there to be the other end of an experiment, you obviously need some kind of test rig that allows you to behave as if you have this long delay component packets in flight, and then you have to be able to somehow simulate the storage component and other aspects of behavior like noise and packet loss. So Mark, how do you construct a testbed to test the idea? Marc Blanchet 16:26 So before that, you've been talking about TCP. TCP was essentially not suitable for space, right? The three way handshake, and the fact that if you need security, TLS, then you add the TLS for handshake, and the fact that TCP is not good in, you know, mobility changing, you know, IP addresses and stuff, and it's in the kernel all this make it, you know, complicated to adapt. One of the things we found when we assess the capability of IP in space is that a few three years ago was we have a new transport, QUIC, [George: right] And QUIC, while it was not designed for space, actually as a lot of check mark, just because it's a modern transport, one check mark is that to establish a connection is only one RTT, right? And that includes the security of it, TLS, right? So only one RTT George Michaelson 17:26 The packet of data needed to create a secure session binding is one transfer, Marc Blanchet 17:32 yeah, one RTT. So with that means is therefore, in that context, where it will take minutes, that is useful, too. George Michaelson 17:41 Oh, yeah, that's made a huge difference already, because it's not a three way handshake across that multi minute delay. The other aspect of QUIC that would immediately say is a good protocol choice is that it uses UDP as its substrate. So where TCP layers directly on IP, quick layers on UDP? Marc Blanchet 18:00 Actually it could, QUIC, could have been designed to run directly on IP. I think they chose UDP because, you know, it crosses firewalls easier than if you have a new number as a transport in in IP, George Michaelson 18:14 so you're not necessarily leveraging any particular aspect, no. UDP, yeah. Marc Blanchet 18:19 UDP, I think they chose to run up UDP because that's, you know, having a new transport directly over IP means that everybody in the path would have to, you know, George Michaelson 18:33 the thing here that I still find confusing is that QUIC, to me, has always felt like it's logistically up in what I call the session layer. It is a transport but it has behaviors that are very similar to the classic network session layer behavior. It has session state and it's agile across variations in transport behavior. I'm not particularly bothered that it's like we're skipping a layer in the stack, but it feels like you've floated up to overcome problems in transport layer. Marc Blanchet 19:04 Yeah, you're right. And for example, we've been, you know, Internet Engineering has been trying to handle mobility right, a device that moves to a different network and change IP addresses, or nowadays, you're behind a CGN or something out of the blue, you're, you're not changing your external IP address, just change port numbers, right? So we define mobile IP George Michaelson 19:32 never, never really took off. Marc Blanchet 19:35 And then nowadays, what we do is, you, we kind of handle this in SDKs and stuff, which is essentially the application layer. But QUIC is what happens with QUIC is it handles the mobility at a transport layer. Therefore IP, you don't need to modify IP, and you don't need to modify the application. So that's the right layer. And because what makes unique the Connection is a connection ID, which is kind of opaque within the QUIC connection, and that connection ID is the thing that remains. So when one of the node is moving, then they just need to recognize themselves with the same connection ID, and you establish right George Michaelson 20:17 and the bootstrapping of end to end security that happens in the one RTT exchange. There are aspects of those protocols that require session re keying, but the fact remains within the lifetime of session key, that kind of key that you're talking about the application specific binding there, you can just use that across this secured session. You don't have to necessarily redo the TLS. Do you you kind of just get everything coming with you for free. Marc Blanchet 20:45 Well, obviously there's one RTT to confirm, you know, the connection, and then there's the concept of zero RTT, but is subject to replay attacks. But you know, just to say that QUIC is modern, more efficient protocol for transport. So when you were asking about simulation, testbed, George Michaelson 21:05 yeah, how do you actually construct an environment to look like this network? Marc Blanchet 21:09 So the first thing was to actually choose QUIC, right? Another feature of QUIC, which is user space, it makes everything simpler, because George Michaelson 21:20 no kernel hacking, required Marc Blanchet 21:22 right? And then you could have configured your QUIC connection for for moon, and then the other QUIC connection for Mars. And so it's very agile for those kinds of environment. Essentially, you can do this with, you know, or four ways. One is you put hardware machines and, George Michaelson 21:42 yeah, I remember people testing high speed trans oceanic fiber by borrowing a reel of 2000 kilometers of fiber and just simply creating a long delay path out of fiber because it allowed them to do lab level testing on terabit comms. But that's kind of hard when you want to do it to the moon, Marc Blanchet 22:03 yeah, and costly and everything. So that's one way going from heavy to light. Virtual machines, Docker, Linux namespace, George Michaelson 22:15 so you're using classic operating system container models that people are familiar with to leverage aspects of their behavior to provide your mechanism for a long delay. Marc Blanchet 22:26 So what happens is between the three, one, the three, the last three. I was listening, I the VM is the most heavy. Linux namespace is the most light, and we actually did both. And we end up doing VMs at the end, there was kind of the guy who pushed that because I wanted to be as near as possible to the real environment, George Michaelson 22:54 right? And so a VM to all intents looks like an isolated operating state. Marc Blanchet 22:59 Yeah, it's an operating system because, you know, sometimes Docker, you know, with networking, and we're kind of, you know, George Michaelson 23:05 they do weird cut through, Marc Blanchet 23:07 right? And I didn't want to have additional concerns about the actual simulation environment, just because, you know, we're tweaking forwardly. George Michaelson 23:16 I think they can mentally see where you might go with this, because when you consider VM environments. What they have to do is they have to put a synthetic construct in between the VM and the real networking hardware and logic of the machine. So most of the hosts which implement this provide a number of facilities to create synthetic networks in the machine and then expose them out into the world. So are you adding a layer of behavior at one of these synthetic network bindings to provide this massive delay in buffering behavior? Is that how you do this? Marc Blanchet 23:52 Right? So you construct so that over time we our solution environment is moved. We did essentially two things. One is where we, as I said, QUIC is really the core thing to simulate and optimize. George Michaelson 24:11 You're delivering an outcome that works in QUIC. Marc Blanchet 24:14 And so when we started at the beginning, we had kind of couple of VMs. And you know, you add delays. use er netem Linux, netem TC- netem, which provides, you know, delays, packet loss. You can simulate everything. Interesting part is you start, you know, you kind of prudent. So you start with, let's say delay 30 seconds, George Michaelson 24:39 yeah. How far do you get on? 30 seconds at light speed in space. Where does that take you to? Marc Blanchet 24:45 You know, four minutes when the two planets are near together, Earth and Mars. It's four minutes when we delay. George Michaelson 24:53 So you're, you're an eighth of the way to Mars. You're talking to a craft traveling to Mars. Yeah. For example, and it's got an eighth of the way there, you're looking at a 30 second delay, Marc Blanchet 25:04 right? So, and then you do this. And then, okay, well, let's try 60 seconds. And then, oh, it works to, you know, you configure QUIC properly here, and then it 300 seconds, five minutes, and net time refuses to actually input the 300th second in the CLI right? It's like, Why can't so, you know, you reach the code, scan a code. George Michaelson 25:29 We don't criticize people who provide software. We depend on Marc. Marc Blanchet 25:33 Let's see, I was not able to find easily a fix I was expecting just, you know, somewhere to change. You know, maximum range, a limit or something. But yeah, anyway, so I contacted the maintainer and told him, You know, I'm trying, and the maximum delay by converging was 272 seconds. And that was on the CLI and then actually, like, in less than a week, if you fix it, sent me a patch, and then, then he put two to the 64 as the maximum in terms of seconds. [George: Wow] Therefore that means you could simulate, you know, many galaxies away, almost the universe. George Michaelson 26:18 We have managed to put devices that are sending some level of signal into the Oort cloud, kind of distance, we're a little beyond that in the model. And so I would think that this means, the higher the delay, the more buffer you, in principle, have to hold because of the potential for packets to be in flight. I mean, we're essentially talking about a stream of photons at radio frequency, energy moving out from a transmitter, either in space, back to Earth or from a ground station heading out towards Mars. And literally, it's a giant hose pipe container filled with photons that are simply going to take this incredible length of time to arrive, Marc Blanchet 27:01 okay, but that's nothing to do with IP. George Michaelson 27:04 That's the lower layer. Marc Blanchet 27:05 They have the same problem today, right? [George: Yeah] so when it sent me the patch asking for, you know, testing it okay, let five minutes. Okay, works. One hour works. George Michaelson 27:16 And this is full flight test all the way to QUIC, [Marc: yeah], coming to a valid completion, Marc Blanchet 27:22 actually, HTTP over QUIC, so I was sending a get request, [George: yes], and then the other peer was sending back the response, which was a file, and then it's like, oh, okay, 24 hours. Well, at that time, 18 hours was the time to reach Voyager, [George: right] So I tried what, let's try to send an HTTP request. Voyager installation, and it worked. It just took so in a few RRT, which is George Michaelson 27:49 you use some tool it doesn't double. You get curl, something that can intuit a quick URL, and you point it into a from a VM to a VM with an intervening system that is using a long delay buffer, and you go and have a coffee or maybe do your washing sleep, you come back to the lab [Marc: right], and you have received the file 72 hours to get the data back. Marc Blanchet 28:17 But that's exactly the same problem that JPL has when it communicates to voyager, right, it sends something, and then you receive the data you know. George Michaelson 28:28 So current technology for deep space probes isn't currently using IP frame. Marc Blanchet 28:35 There's no network in space correctly. George Michaelson 28:36 So the framing that I understand they use is very, very carefully designed error correcting codes and also replication of data. So they might do something DECtape used to do, and they might send the same command sequence three times, so that you can do a two out of three weighted sum, and those kinds of things. But you can also do reconstruction of the data flow, because you have massive overhead of error correcting, which is time expensive if you're talking about terrestrial networks, but if the delay component is measured in minutes or hours, it's worth investing the time, right, Marc Blanchet 29:13 right? Yeah, so the link layer is actually the standard by the equivalent of IE TF for space, which is the CCSDS, consultive committee for space data systems. And so they do codecs in pipelines. So two codec in a pipeline way. [George: Yeah], and the reason for this is that when they send commands to the rover or to Voyager, you don't want you know the command to arrive with errors. George Michaelson 29:42 So, so there's no command received. It's one thing, but a command that's wrong is a disaster. Marc Blanchet 29:49 Could be a disaster, therefore, which is surprising, because in, you know, in normal thinking, people would say, you know, space is hard therefore the signal to noise ratio is probably bad. George Michaelson 30:03 Yes, I was, I was expecting, Marc Blanchet 30:05 right? And then, you know, given that the power diminished to the square of the distance. You know, when you talk to a voyager, it's far away, right? So the power meaning at the end is pretty low, and all what I just said is right, but with the what they do with error correction means the output of this is when you talk to mission and I talked to mission operations in.. for Nasa for European Space Agency, they both told me exactly the same thing I was asking them, not at the bit level, but that the frame level. George Michaelson 30:45 So after, after all of the error correcting capabilities of layer one layer, Marc Blanchet 30:50 what is your error? Frame error rate? [George: Yeah], say zero. We're not looking at this. Maybe you have seen on on TV, you have, or on pictures you have their the big mission control systems and lot of the screens and stuff, there's none that has frame error rate, which you know, you and me are, you know, a typical person would say, George Michaelson 31:15 whereas the three GP class service that you were talking about, that was sent Up to the moon that's actually operating in a much more terrestrial error rate. That's a protocol, because the delays are short, that it's okay to have loss in the system, because you can afford to perform the recovery we're talking here with a very, very long delay. You really badly don't want that. So the framing, the link layer, aims to be as error free as possible, and it's only that the higher protocols, be they QUIC or whatever, need some way to say okay or to emit the stream of data that you're expecting to receive back, Marc Blanchet 31:53 right? So you should not count on the fact that the frame error rate is zero. But what it says is you don't need to design the system with a lot of packet loss, right, [George: right] So that makes you know your considerations were designed differently. George Michaelson 32:07 So were you able to model the idea of the intermediary system? Because some of the stuff that I believe you've worked on in delay tolerant networking is deliberately designing things that look much more like old fashioned, old school UUCP store and forward, were you able to model that kind of behavior? Marc Blanchet 32:27 So yeah, what we did is, I said three things to make it work. One is storing packets. You know, when there's no communication window with the destination profile, QUIC and applications. So back to, for example, the simple test of HTTP Get - you profile QUIC. HTTP doesn't have any notion of time, so that's kind of no brainer. The remaining is storage of IP. So what we did was a prototype. I had one of my guys who has been doing you know kernel kind of thing for the networking stack. And I asked him about, what do you think about, you know, this IP storage packet, sorry, difficult thing. Did it in a week, and came back few 100, like 500 lines of C code and done, obviously like a prototype, but still working very well. So what you do, essentially is you use a TUN interface on Linux, which is a kind of, essentially a virtual interface, George Michaelson 33:35 yeah, which breaks out into user space, no matter where the packets come from. You then have a stream of bytes, just as if you were on the network wire, but it's in a program, Marc Blanchet 33:45 yeah. And then, therefore, you essentially configure filtering that all input packets coming from to the node. You just tell the to send it to the TUN interface, and then the TUN interface, it does very simple thing, which is, okay, you couldn't make it smart or not smart. We decided to make it not smart. Which is what you do is essentially say, Okay, I received the packet in the user process, and then what I do is, I just send it back to the kernel. You know, please forward it and the kernel. If it is not able to afford it. That's because the destination route is not there. It tells you that, and then, therefore, oh, okay, there's no destination route. So store it, you know, George Michaelson 34:29 store it. Don't reject it, [Marc: right], right? So it changes behavior, but it doesn't actually alter the end to end protocol semantics. It's like a naughty, non compliant intermediate router that doesn't do what most people expect. [Marc: Well, it's not compliant.... But] okay, maybe that's the wrong word. But if the normal behavior is when destination unreachable, drop, this thing is when destination unreachable, store, Marc Blanchet 34:55 yeah. And then, therefore what you do is then register to netlink. Events that tells you when, for example, you could, for example, register when interfaces are going up or route are inserted in the forwarding table one of the other then what you do? As I said, I wanted something not smart. Whenever that kind of event happens, you just take your queue of packets and then just try to send it back to the kernel, and if it doesn't forward, then you start it again. George Michaelson 35:26 So if you think about the reality of a system in space, you have these radio links, and their layer two construct is this highly reliable encoding of data. But they also have effectively sideband information control channel information, like signal to noise ratio, or loss of signal. So a thing in space is going to say, Well, it's been occluded by the planet. I have loss of signal. You sit there and you say, Fine. And you wait for the orbital delay, and then you get a signal that says, signal re establish bit error rate dropping, frame rate rising. We have link, and you then say link, bingo, and you send out packet string, Marc Blanchet 36:05 right? Because the IP stack, the event that you see is essentially an interface up, right? That's the reality, right? Exactly. So everything behind quote, you don't care, therefore, we didn't do anything smart, but you could make it smarter, right? For example, we did more like a FIFO approach, where, when a link is back, we just take all the packets in a FIFO, FIFO approach, and then try to forward them. You could classify them based on various, you know, criterias you could have. And that's fundamental core issue. Research issue is, if you are aware of because you know when the satellite goes, I mean, this is all planned and known, right? George Michaelson 36:56 Orbital mechanics are really well understood. Marc Blanchet 36:58 And not only orbital mechanics, but the actual time where the community communication window with between the peers could happen, if you have that knowledge, you could be smarter, right? Because then you know who you will be talking to, and things like that. George Michaelson 37:14 You can construct packet trains that arrive to the size of the buffering capacity sit there, and at the point you get resumption and you're draining, you can be backfilling in the knowledge you've emptied the buffer, and it's capable of receiving the next incoming stream and handling and so you can kind of optimize the packet flow through this thing, Marc Blanchet 37:34 yeah. And then you know that in five seconds or one second, that temptation will go down. Therefore, you know, you could do smart stuff so, but we haven't done so with that. We could create the simulation environment, and nowadays evolved with a UI, and then you create a JSON file which describe the topology, you know, the nodes, and then another file with the nodes and the links, the links include characteristics such as delay, and then all the things you want to simulate, packet loss, or, you know, whatever, reordering all the net and essentially, you know, push to netm. You know, do this for me, please. George Michaelson 38:18 This isn't magic, and there are going to be pathological cases which cause quick, a lot of extra delay to re recover through the behaviors. It's not that anything is free here. The point is subject to tolerance for delay. You appear to be saying the protocol survives the time-independent aspect of this protocol, it's quite easy now to demonstrate really, really long delays well beyond foreseeable science experiments in space. So radiation hardened CPUs tend to lag current CPUs by a couple of decades. But nonetheless, we are talking MIPS, MIPS class. That's Marc Blanchet 38:58 That's normally the case. George Michaelson 38:59 That's not the case anymore. Marc Blanchet 39:01 No, that's something unrelated to the whole discussion we have, but the space agencies have been so what happens is the they used to procure specialized hardware, right? Yeah, hardened for radiation and George Michaelson 39:18 wider tracks, all of Marc Blanchet 39:19 all those things, but then they got into a problem. It costs a normal leg, a single board could cost you a million dollars, like , and moreover, it actually doesn't guarantee much, because this is done almost manually. Therefore even if you test one, the next one may not be working because it's kind of a manual or almost manual stuff. So in fact, you're trying to have something very, very good, but it doesn't, you know happen. George Michaelson 39:52 So in fact, we don't have to worry about the capability of a space bound CPU to perform complex error correcting, And modern protocols to have enough local cache to run the stack. It's just it'll run a VM. It's going to work Marc Blanchet 40:07 all right. And so nowadays they use essentially off the shelf stuff that is premium, you know. And then what they do is essentially test them in the lab with all the radiation that they could then when one CPU or board is not behaving, just, you know, throw it away or actually use it for your other stuff, and then you keep the one that works. So even that, for example, one of the orbiters around Mars was sent in the early 2000 all the orbiters around Mars are now used as relays, communication relays, George Michaelson 40:43 right? They have an ancillary function, Marc Blanchet 40:45 yeah, which was essentially observation of Mars, right? They were taking pictures at various wavelengths of Mars, George Michaelson 40:53 but they can act as a relay for the on Mars, ground based surveying devices, and provide a higher bandwidth link locally, and then they have space bound antennas, and can be doing signaling on the long path to Earth Marc Blanchet 41:07 Exactly. So they repurpose them or add that functionality. But then, for some they essentially don't do any more observation. So what happens is, for the purpose of observation, because those orbiters were going around the Mars, therefore they could not send, you know, data back to Earth, because it was not possible. Yeah, they had storage, right? So the early 2000 orbiters have like, 160 gig of storage, you know, because of you want to store images, right, which take this Some, yeah, so as a communication, really, you have quote, "free" 160 gig for something that was, you know, design, design in 90s, George Michaelson 41:56 nothing in space is free, but for the context of experiments with doing DTN delay tolerant networks, layered IP experiments, QUIC with long delay. The technology that you know is viable for space, you also know is viable for a potential future IP platform. Marc Blanchet 42:14 Well, I'm not sure we will actually run IP we could. But my point is, it's easy to actually have good storage in space, [George: yeah], especially with, you know, off the shelf stuff and solid state memory and stuff like that. However, you can be dumb in a sense that there is no point of, you know, sending a 4k video from moon to Earth, you know, full bandwidth. If what will happen is then, which would require a lot of storage, very fast. Therefore, you would orchestrate applications and network so that you use the overall infrastructure somewhat optimized, right? [George: Yeah], you know, if you're dumb, you'll just fill out the buffers, and then when the buffer is full, George Michaelson 43:06 there's a time for being dumb. But if you know exactly what you're doing, you will send lower bandwidth state to maintain context, and up the bandwidth and the encoding when it's interesting and turn it back down again. Marc Blanchet 43:18 Yes. So therefore, in some ways, you want to have big buffers to avoid, you know, packet loss as a standard IP forwarder, which is no destination drop the packet you want to avoid that, therefore you need buffers, but at some time, you don't want to use them so much that they will always be full, right? So you need to know it's for adults. George Michaelson 43:45 So Mark having shown with simulation and earth bound test beds that you actually can have highly delay tolerant behavior in a QUIC based protocol. Do you have vision of maybe something in the foreseeable future when we're going to be doing tests in space, perhaps in low orbit or in lunar context, is that coming up? Marc Blanchet 44:07 Yeah, so we're working with Spanish research institute that does satellite right now, and we're working together and using it for the purpose of testing that would exercise the intermittence, because there's only one satellite, therefore we will not be able to communicate all the time. It's not a constellation, but the delay will be LEO delay, right? So that won't test the George Michaelson 44:34 but it tests part of the model, and it also tests in space. It's not an earth bound experiment. So that's your first level test. Marc Blanchet 44:42 Yeah, the other test that will happen is more as partners of space agencies that want contracts for communication release for moon, that's the next step in real space. And for moon, one thing. That is, you know, interesting also is that for European Space Agency, the orbiter name is called Lunar Path Finder. And that one, they decided to be a layer two forwarder. So what that means is, will act like a switch right from the point of view of IP, right? So therefore they will do store.. store and forward, but at layer two, which means that that device may not need like a switch, unless you manage it by IP. May not need to run any IP stack, right, [George: right] It's just at the layer two. George Michaelson 45:37 But if there was an opportunity to test the field Test activity targeting lunar bases, which are foreseeable science experiments, and then potentially permanently occupied space. I mean, that's a long way out. These things become interesting, Marc Blanchet 45:51 yeah. But even if it's layer two, you will be able to test end to end using IP QUIC, but that device, intermediate device, will just not do IP packet, packet storage, but layer two frame storage, right? George Michaelson 46:07 So different mechanism, but a similar outcome, Marc Blanchet 46:10 but the same, the same overall. The only difference there is that the forwarder is smart because it doesn't have knowledge of, you know, you cannot do ip route doesn't have a notion of IP routes and stuff and alternate path, and so it's, you know, layer two. So, but that's fine. I was trying to say that you could have different deployment models, and still the the overall architecture works. The other thing that is actually done is that there's a current orbiter that China sent over around the moon, and they put a QUIC stack on it, and they actually started to, like, last year of test with that orbiter. George Michaelson 46:53 So there are other research groups looking at this right this area. Yeah, that's really great. Thank you, Mark. That's fascinating. Can people read about this at the Virginia site? Marc Blanchet 47:04 Yes, somewhat we do have, I think two resources is so there's now, since a few months now, well, a bit more than a few months, there's a new working group in the IETF called TIPTOP, means taking IP to other planets. So that's where we do. You know, the standardization, which is, at the moment, not new protocols. We're actually defining, oh, you would run the IPC George Michaelson 47:34 profiles rather than inventing new things. [Marc: Exactly], we'll make sure there are links to that, and then blog Marc Blanchet 47:40 before the working group, as I said, we did a lot of simulations and assessments, and we look at NTP behavior, time synchronization is very important in space, [George: yeah], but it's very difficult, [George: yeah]. So we look at, you know, streaming and all kind of stuff. And so what we did, we had multiple IETF where we were having informal meetings. And so what we're heading into is is we have a whole set of slide deck simulation tools that we open source. You know, all this is on GitHub, [George: right] Is then the organization is called Deep Space IP. So single word, DeepSpaceIP.GitHub.com github.com George Michaelson 48:25 We'll make sure a link to that Marc Blanchet 48:27 thing we have done there, and we continue publishing the stuff there. George Michaelson 48:31 That's really great. Mark. Thank you very much. Marc Blanchet 48:34 Sure. My pleasure. George Michaelson 48:36 If you've got a story or research to share here on PING, why not get in contact by email, to ping@apnic.net or via the APNIC social media channels. Also remember the measurement@apnic.net mailing list on orbit is there to discuss and share relevant collaborative opportunities, grants and funding opportunities, jobs and graduate placings, or to seek feedback from the community on your own measurement projects, be sure to check out the APNIC website for All your resource and community needs until next time.