Job Snijders 0:00 And I knew the rsync protocol from my system administration experience, and I have used rsync in many contexts to make backups, to transfer files, and I like this utility so it feeling like it's not performing as well as it should be, or could be, in relationship to the data. Yeah, at some point I was questioning myself, but so let's dive into why I think they are somewhat inefficient. And in the case of rsync, what is very efficient is that it transfers the difference on the block level between now and, I don't know, like 30 minutes ago or an hour ago, but what is very inefficient is figuring out what changed. George Michaelson 0:52 You're listening to ping, a podcast by APNIC, discussing all things related to measuring the Internet. I'm your host, George Michaelson, this time on ping, I'm talking to Job Snijders. We caught up at IETF 125 held in Shenzhen, job presented to the SIDROPS Working Group on a new protocol he's been designing called Erik the Erik protocol was named in honor of Erik Bais, who died in May 2024 Erik was a stalwart of the ripe routing community. He was chair of the address policy Working Group and active in the Dutch cloud community and the Data Center Association RPKI, the principal mechanism for determining secure inter domain routing intent. Hence, SIDR depends on every relying party validating the data and collecting all the signed statements from all the publication points worldwide. This is a time consuming process which inherently serializes as a sequence of bytes fetched to form a given repository state at a given publication point and how the protocol works out what has changed since the last fetch by this user and what to send causes this serialization. It's not very efficient, and it's not scaling as well as we'd like, given the amount of data rising and the number of validators or RPs who are now fetching the data. Job's. Erik protocol is designed to improve significantly on the two mechanisms defined at present, the rsync protocol, originally designed in the mid 1990s for file system synchronization and RRDP, a SIDR specific "delta" protocol, which was designed to improve on rsync, using experience gained from implementing the NRTM mechanism, use the copy data in the RIR Whois databases. Job, welcome back to ping. It's lovely to talk to you again. Job Snijders 2:54 Thank you so much, George, George Michaelson 2:55 So you are very active at the moment in RPKI. You've been doing some really interesting work in the mechanisms we're using there, which kind of does relate to activity you've also been measuring. This is, if you like constructs coming out of the measurements you've been doing on the behavior of the system. Do you think that's fair? Job Snijders 3:14 Yeah. Or phrased differently, I develop RPKI, software that is used by many ISPs around the world, and part of that development process is observing how does my software behave in the wild with the real RPKI distributed database that exists on the Internet. And this basically means that every day, I wake up and I run my programs, and I look at the outputs and I observe what is the program doing? Now, at some points, I had a feeling creep on me, that whereas, like it is, it is taking so long for the process to complete. George Michaelson 3:51 Yeah. So just to give people who may not be deeply embedded in RPKI a sense of this, this is a public key infrastructure data model where all of the people who are involved in signing over their routing intent are making signed objects called ROAs that declare their intention for origin-as in BGP routing, and a new emerging structure called ASPA we might talk about later, but the point is to perform the cryptographic validation in this information space, you have to get everything. And alongside this, people are free to choose where they publish. And alongside this, for many, many people, the process of fetching data has become essentially serialized. I go and look at place A and then I go and look at place B. And so you in the software development space, you've been observing the validator that you've written, RPKI client, getting slower and slower and slower to arrive at the end point, haven't you? And you've been kind of going back in saying what's going on here. Job Snijders 4:58 Well, slower and slower and. It depends on the day. It was almost like observing the weather, like, how fast is the system today? And as I observed that, yeah, over time, the database is growing so George Michaelson 5:09 so that's like a given. It gets slower because it gets larger, Job Snijders 5:12 right? So if we compare the size of the database between now and, say, five years ago, the database is now five times larger, give or take like, depending on how you look at it, but it's substantially bigger than it was a few years ago. And the slowness of the downloads, it surprised me, because the protocols that we use today are advertised as efficient "change only" protocols that should perform well, George Michaelson 5:40 yeah, so these protocols are actually quite interesting. One of them is almost an undocumented historical artifact of a past time. It's this protocol called rsync, which was designed by a lovely guy in Canberra in Australia, Andrew Tridgel, and it was based on the idea of examining Unix files, block by block, and only sending the blocks in files you could detect had changed. And if you consider constructs like log files and other append file structures, it was really beautiful for this, because it would basically just do tiny checksum exchanges, ignore the front part, and then send the blocks that were the growth of the file. But that mechanism is an archaic model of code that's been brought into the lifetime of everyone here since about 1990 the newer protocol that we use, RRDP, it was designed, actually, in part because of work people like you had done designing this protocol called NRTM, which is used to propagate changes in Whois it was designed to be a "delta" protocol, a mechanism to explicitly say, if you know what you've got and you tell me when I got it, I'll give you the things that have changed since you got it. So you're absolutely right Job, both of these protocols are designed to be efficient in air quotes that you can't see on radio in that data model, but please continue you're observing they're not as efficient as we think, Job Snijders 7:09 right? And I knew the rsync protocol from my system administration experience, and I have used rsync in many contexts to make backups, to transfer files, and I like this utility so it feeling like it's not performing as well as it should be, or could be, in relationship to the data? Yeah, at some point I was questioning myself, but so let's dive into why I think they are somewhat inefficient. And in the case of rsync, what is very efficient is that it transfers the difference on the block level between now and, I don't know, like 30 minutes ago or an hour ago, but what is very inefficient is figuring out what's changed. George Michaelson 7:51 Yes, there's a pre state in the protocol, right? You tell it the base of the object set you want to get, typically, a file system anchor point, a mount point in the file system. And it has to descend the entire tree of state and build a map. Here are all the things in this state. And the client has to say, got it, got it. Got it, got Oh, not that one got it, got it, got it. Job Snijders 8:15 And, you know, the data is these days, host on memory file system or SSD, so the IOPS part is, at least on the client side, is not a prohibitive aspect. You can walk a file tree very fast, but the RPKI nowadays is roughly 520,000 tiny objects. George Michaelson 8:33 Oh, wow. That's huge Job Snijders 8:34 with a median file size of two kilobytes. And some of you may recall that if you have a large directory with many, many, many tiny files, moving that around on your computer becomes tedious. George Michaelson 8:47 That actually incurs costs in the design of the operating systems, model of how much it has to hold in state in memory against the catalog of files in a directory you exhaust that buffer in kernel walking into directories that are large spaces. And rsync is a beautiful protocol for doing block differences, but it has to interrogate the file to find the block differences. And if all your files are tiny, this is a hugely inefficient cost. It's basically asking tiny objects have you changed? Which begs the question, How often do tiny objects in the RPKI ecology change? And I have a feeling Job re-signing means that quite a lot of these objects change all the time. Job Snijders 9:33 If you look on a day to day basis, about 180,000 of the objects change somewhere in a 24 hour period. George Michaelson 9:42 And these don't form a nice, regular sub pyramid of this scheme. They're distributed throughout the 500,000 object Job Snijders 9:50 set every second, two objects change somewhere on the planet, and we don't know where they'll change on the planet. But even if we ignore the descent on the file systems. Let's ignore the entire file system. Just look at the rsync protocol. Sending you the list of the files has become prohibitively expensive, because a list of files in the rsync protocol to cover, say more than 500,000 files is around four megabytes, George Michaelson 10:20 right. Job Snijders 10:20 So that means that every time I connect to the servers and nothing changed, I am still receiving four megabytes of data. [George: right], the most expensive API call imaginable. George Michaelson 10:32 and it's not a single packet round trip to be told, four Meg, this is sent over higher protocols that involve SYN ACK sequences. They're inherently a serialized stream from that source. So you're doing four megabytes to fetch that data every time, possibly to be told, nothing to see here. Move along. Job Snijders 10:53 So I connect to the rsync server. I'm like, Hey, George, what do you got for me today? George Michaelson 10:57 Same stuff I had yesterday? Job Snijders 10:59 No, I wish you would tell me that what you're telling me is the full inventory of everything you have. George Michaelson 11:05 I got sandwiches, I got soup without sandwich, I got coffee, I got fruit, soup and fruit, or the option of the coffee in the soup, Job Snijders 11:13 right? And I'll be like, Oh George, I have just eaten all of the but I'll be back in five minutes George Michaelson 11:19 and asking me exactly the same relationship Job Snijders 11:22 over and over and over. So my realization with rsync was the transfer is highly efficient. You know, it's hard to beat Sure. You know, it's great, but figuring out what changed that is the expensive part, and it is becoming a blocking factor to scale the RPKI to the dimensions that we need for future generations. George Michaelson 11:43 So RRDP, the Delta protocol, informed by NRTM and change Job Snijders 11:51 IXFR George Michaelson 11:52 And DNS Ixfr was an intercept to a more efficient transfer style, where, if you think about things like MPEG encoding, where you have I blocks and P blocks, the index blocks and the changes. This model assumes you periodically may incur a cost of a massive synchronization, but you're then given a set of deltas that you can apply. And I want to believe we did an okay job designing that. And I want to believe that should be more efficient, but I'm looking at you thinking you're about to tell me it doesn't do a good a job as we want. Job Snijders 12:28 Well, it does something that I'm not sure that was fully understood in terms of consequences at the time of design. So let's walk through why RRDP appears to be efficient. The idea is that the publication engine writes into a journal, and you download the journal in segments, the deltas and on paper, this sounds great, because between the last time that I synchronized and this time that I'm synchronized, the server will tell me, Well, three new segments or three new deltas appeared, and I download the three deltas, and we're good to go. And these deltas contain the binary objects base 64 encoders, wrapped in XML. And originally I thought these layers of encapsulation are what makes it inefficient. George Michaelson 13:16 It does sound like over encoding of the fundamental data, so it's a reasonable thing. That's why it's slow. But it Job Snijders 13:23 It wasn't because the moment you run this through any compression, gzip for instance, it compresses down to levels where it's like the encapsulation is non existent, George Michaelson 13:35 right? Because the framing components, textually of this stuff appear as level as if compressed George Michaelson 13:41 the tokens. Job Snijders 13:42 That was a red herring, right? Okay, but why does it feel so my realization with RDP is that RDP was designed for full history replication, and on paper, this sounds great. You have a validator you want to synchronize to the database so you replicate the full history of the database, [George: sure], but you don't need the full history. The RPKI is churning so much that in the span of 30 to 60 minutes, a lot of events have taken place that have overtaken previous events. So the inefficiency in RRDP is that you're always downloading data that has become irrelevant. George Michaelson 14:22 Yeah. So there's a component of mathematics, the concept of partially ordered sets and sequences in things like MQTT queuing systems, this kind of model of message bus, where, if you consider an object A and it's created at time t and then at some future intervals, t plus 1t, plus 2t, plus three, you add a field to A, you double the size of A, you regret it, and you shrink the size of A. And the very last thing you do is you delete A that partially ordered set of A, modify, modify, modify, delete, if you're asking what. It's the current state. The current state is delete. You don't need to know that prior history if all you want to know is it doesn't exist. And it feels like you're saying this protocol had no concept for any object, of constructing the partially ordered set and reducing it to the minimally necessary components to arrive at its final state. It rehearses all the history to get there. Job Snijders 15:28 Yep. So if we make a weird analogy, you want today's newspaper. So you're like, hey, Job, can you get me today's newspaper? And I'll be like, Sure, George, but you have to accept yesterday's and yesterday's yesterdays newspaper as well. And us clients are like, but I have no interest in yesterday's news. George Michaelson 15:48 Yeah, that's under the cat food. I don't need that, Job Snijders 15:51 right? And I'll be like, I am so sorry, but you have to download yesterday's newspaper. There is no way to not download it and arrive at today's newspaper. George Michaelson 16:01 So we have a protocol designed to be more efficient than rsync, a protocol designed around the models of Delta. It has an encoding overhead that you've been able to demonstrate from analyzing its behavior disappears under compression, and we're left with it's sending the maximal set of change history when all it needs is the final state Job Snijders 16:21 exactly! George Michaelson 16:22 Well, it would be lovely to imagine we could re enter both of these protocols and optimize them to achieve a better outcome, but I have a feeling the rsync community don't particularly feel this need because their needs are met in rsync and equivalents specific to synchronizing file systems, because that's what rsync is designed to do. And RRDP was designed as the better protocol, but it's manifestly not fit for purpose. So my gut feel is we start again and try and invent something a little better. So tell me, Job, did you come up with a better way of doing this? Job Snijders 17:00 Yes, sir, I did. And not only do I think that the new approach, which is called Erik synchronization, is better in terms of the downsides that we discussed just now, but it also addresses an issue that rsync and RRDP share with each other in the way that the repository distribution was arranged in the RPKI data structure. Each publication point has a monopoly on access to its data. So this means I download APNIC data from the APNIC RRDP or rsync service, and I cannot connect to the ripe server to get APNIC data, George Michaelson 17:41 you know... Job Snijders 17:42 and this George Michaelson 17:42 we did talk about this in hypothesis during the design phases. I should say at this point that although my current life is to be a talking head in a previous life, I had a role to play in the design of these kinds of systems. And we talked about the idea it would be nice if some neutral point could hold and publish all the data we want. Job Snijders 18:04 Yeah, George Michaelson 18:04 but we've encoded in these cryptographically signed objects reference points to this is where you go to get me. And we didn't think about, how do you map from that canonical reference point to an intermediate agent who could probably supply it to you. Now, intermediates will raise a lot of hackles, but the thing is, remember, everything in this system is cryptographically signed, so there is no real downside to being told it by a third party. You are required to check it before we move on Job. Am I right in thinking you call it Erik as a memento mori of someone in the Dutch research and routing community. Job Snijders 18:44 Yep, the protocol is named after my friend Erik Bais, who unfortunately passed away in 2024 George Michaelson 18:51 he was a quite significant member of the community, wasn't he? He took lots of interesting initiatives. Job Snijders 18:57 He was a big fan of routing security and a promoter of RPKI, he hosted the very first RPKI nodes to help distribute RPKI data. So at some point it's clicked. I was like, if I'm making a new protocol, it must have a name. I'm not going to go for a weird background name. I will just ... reference my friends. George Michaelson 19:17 So we have a problem. The architecture requires a fetch from the originator, and we have no conscious understanding of use of aggregating intermediaries or any structure that might permit us to understand the overall shape of the information space. So you put this on the table as one of the goals you wanted to fix Job Snijders 19:38 Right The goal is we solve some problems common to rsync and RDP we solve and we take the best parts from rsync and the best parts from RDP. And in my mind, if I look at the attractive parts of RDP, it runs over HTTPS. It's the lingua franca of this community. George Michaelson 19:57 It's the new waste in the hourglass for. Working protocol development. Job Snijders 20:01 So we want something over HTTPS. We want something that allows you to jump beyond the historic events that no longer matter. So that is the component from rsync. We want something that doesn't require an active process on the server side, so that it's all static data, which is and property of RRDP George Michaelson 20:22 Yeah, there were efforts to modify rsync to include pre computed catalog models of data fetch that was actually really important back in the usenet UUCP days. But the thing is, they're always adjunct services, and if you're designing a protocol to replace it, you should build the behavior in day one. Job Snijders 20:40 Yep, I consider an unwelcome complication in RRDP, the fact that an XML container was used because it meant that all the RPKI validators had to link in an additional software library dependency to be able to interact with the protocol. So I tried to make the protocol reuse components that we're using anyway in this ecosystem. So the messaging layer in the Erik synchronization protocol is ASN.1 or distinguished encoding rules DER, because that's what validators already talk George Michaelson 21:15 gotta say, this is my least favorite protocol. But if you're a telecoms engineer, ASN.1 is a piece of IBM era, historical dependency of bit efficient on the wire encoding in a context. So in ASN.1, you receive two bits. You know their meaning because you were told in a schema what to expect in this two bit sequence, given things that have happened before, and it is tiny on the wire. It's efficient. Job Snijders 21:44 It is not only efficient, but it gives you canonical encoding forms and canonicalizing What the on the wire, shape of the bytes is, means you have item potency, and that opens your door up to distributed computing solutions where you have multiple writers, multiple readers, all doing work concurrently, and you can build redundancy into the production and consumption of the George Michaelson 22:09 data, because the data stream is absolutely predictable based on the condition it was in at the time it was generated, and expectations of behavior anyone else can do a reproduction, they know exactly how to perform the encoding to match Job Snijders 22:24 right. And the reproduction in the Erik protocol, it happens on both the clients and the server side, and both encode towards a Merkle tree shape, [George: right] And then you can do the comparison to figure out, what is it that changed? George Michaelson 22:40 Now I don't want to do a deep dive down structural behaviors, but Merkle trees are interesting. They're the product of someone who I think we can say is genius class. Job Snijders 22:51 Oh, absolutely, George Michaelson 22:52 in information management, cryptography, design of systems, and they've become very important in a space I hate, which is Bitcoin and that world of crypto currency, but it's important to distinguish the information structure from the horrible behaviors in Bitcoin and crypto currency. The information structure is fascinating, Job Snijders 23:15 yeah, allow me to quickly summarize how a Merkle tree works. Imagine you have 10 files on offer, and I want to have those 10 files now, the most naive way to replicate the data would be to just ask you, George, send me the 10 files, and then I have downloaded 10 files. Now, if one of them changed, how do we efficiently figure out which one of the files changed? And a possible approach is you hash a SHA256, message digest over the 10 files, and you have 10 files, so you end up with 10 hashes. George Michaelson 23:50 So each file gets a hash. George Michaelson 23:52 Yeah, you sent me the 10 hashes. I calculate the hashes on the 10 files I have in my match George Michaelson 23:58 and if one of them matches and one of them doesn't you've identified the changed file, Job Snijders 24:03 right? But you sending me 10 hashes, still is a little bit inefficient. So what if you create a hash over those 10 hashes, and you sent me that one hash, George Michaelson 24:14 right? And so if you now scale from 10 files to 100 files, but imagine i hash every 10 files, I can only send you meta hashes, and you get to detect one of the block of 10 is to be considered. And you can decide the size of that meta footprint and how effective the hashing calculation is as a function of your sense of risk of change inside this framework Job Snijders 24:37 Right! So in the Erik synchronization protocol, the Merkle tree is three phases deep, and what it boils down to is I can ask the intermediate the relay for what is the latest hash over the top of it, and I get one hash. And if it's the same as what I computed over the data, I have locally, then we're done. Then my exchanging just a few bytes, I have figured out that we are synchronized. George Michaelson 25:04 So a point here is the hash is a calculation that is valid against no change, which means if you compute the hash on change and store it, when you're asked, What is your hash, you don't have to calculate it. And so both this intermediate and you as a client, if you compute your hash in detection of change the minute you decide to do the game again, you can fling the hash and check the hash. Nobody has to do sums just because a client comes to the door. Job Snijders 25:35 And if the hash is different, I will fetch the hash that you gave me. And the trick in the Erik synchronization protocol is that you you don't fetch the objects by name. You fetch objects by hash. So I asked the web server give me the object that belongs to this hash, and that means that I have integrity verification on the object contents, because the hash of the object is the URL, so to speak, and then I fetch the data that changed, and I can walk the tree to figure out where in the tree are the objects that change. So this is very, very different from both RRDP and rsync, because the client knows a-priori what it wants to fetch from the server, whereas in RRDP, it's a little bit more like a surprise. George Michaelson 26:23 Yeah, tell I know something's changed. Just give me the changes. Give me where's this model? You have a 3d hashing model that directs down to the smallest set you'll wear the cost of getting to then find the specific change. And you have moved from a namespace of canonical fetch points which are encoded in objects in RPKI, to hashes which are constant length, objects constant size, constant cost. You've taken variable cost because it's potentially pointing at indirect places that require you to connect elsewhere. So you've actually done two things, Job: you've constructed an information model that is indexed on hashing, and you have intruded the concept of an intermediary that is potentially able to tell you this across many back end sources. Job Snijders 27:13 Yep. And why this all matters if we look at ROA propagation, the ROA is a very powerful utility, right? It is a very high impact facility that we use for good to improve routing security. But if you create a ROA and you mistype the origin ASN, you might disrupt service for the customer whose prefix is encompassed in the ROA. So we're sort of moving away from the consequences of a typographic error in the BGP routing protocol to there being consequences if you make a typographic error in the creation of your ROA. George Michaelson 27:52 And it should be said that, again, reaching back into the past, we'd imagine cryptography was about making the longest possible signing assertions to reduce the burden of signing. And what we forgot is an object is valid for the lifetime of its certificate, and there are risks in long lived cryptographic signed objects. Some of them, yeah, it's beneficial to live a long time down at the edge where things have to change and you may make mistakes of consequence. You actually want short lifetimes, which means you want the ability to say, I need that to go away and have people be responsive to it. People need to look at these things more often than we thought. Job Snijders 28:33 You're totally right. So let's, let's run through a scenario that could happen to anyone. Your boss tells you, hey, RPKI, I heard good stories about it. Can you make a ROA? And ROAs propagates on the Internet in the span of between, say, 10 and 120 minutes. And P95 is a little bit less than an hour. So you could say, you know, on average, most of the internet will have seen your new ROA within an hour, but it also could be as short as 10 minutes, if you're lucky, and if you're a little bit unlucky, it could take longer. So you create a ROA, and after you hit submit, you realize, oh, boy, that was not optimal. I copy paste it, and I left off the last digit, but the ROA you created that contains the error propagates in 10 minutes, and then you quickly fix that ROA. But that version of the ROA takes two hours to propagate, and that means that for 110 minutes, you're kind of fiddling your thumbs, hoping the ROA will start propagating, but it's out of your control. And this is why I think it is critical for this community to really engineer for optimal performance. We have consistent performance, predictable performance. ROAs must propagate as fast as they are created, and I think the Erik synchronization protocol is a step towards predictable propagation times. George Michaelson 30:01 So here we are sitting in IETF 125, in Shenzhen, and something that's come up as a discussion item in the corridors is the idea of code-first, standards development. And I know from other conversations that you've been promoting the idea for work to be taken seriously inside SIDROPS it's important we get to working implementations beyond conceptual proof. This isn't a paper design is this Job? Job Snijders 30:29 I am happy to report we have running code. And it is really cool to be thinking about ideas and what could work in practice, but it is much, much more interesting to verify that the ideas do in the work in practice. George Michaelson 30:44 And so you now have some measurements using real world data to demonstrate what it would look like fetching the world of ROA through intermediaries, using Erik Job Snijders 30:55 Right, my preliminary measurements suggest that, depending on the refresh cycle of the validator. So a validator could, say, synchronize every 15 minutes or every 60 Minutes. Like let's take those two patterns as an example. And if you synchronize a validator every 15 minutes using rsync, it is quite costly, because, as we discussed, with rsync, you receive the entire menu of everything that's on order, and then you exchange differences. And if you compare rsync to RRDP on the 15 minute interval, RRDP is more efficient because there's no giant file list in RRDP, but if you synchronize every hour, because with RRDP, you're pulling in the full history of everything that changed George Michaelson 31:42 And across the surface of this space. Some things have gone through multiple changes. You're pulling more data than you need, Job Snijders 31:49 right? So, weirdly enough, if you synchronize every hour, rsync is more efficient from a bandwidth consumption perspective. And these results surprised me. I was like, That is counter intuitive! George Michaelson 32:00 enough, we know a kind of range of interest, 15 minutes and an hour, two protocols with a weird outcome. How does Erik play in this space? Job Snijders 32:08 My initial measurements suggest that Erik synchronization performs better in both scenarios. And this is really good news, because for an RP to decide which of the protocols should I be using, and basing that on the synchronization frequency is a massive implementation complexity. So having a protocol that, regardless of the timing interval, performs better is a very, very desirable thing to have, because it means that whatever schedule you're on, Erik synchronization performs better. George Michaelson 32:40 So you mentioned that it's HTTP is a substrate. This makes me immediately ask, Is there potential for the CDN community to be able to consider themselves as the intermediaries and provide us with a huge surface of reliable, short path visibility? Because one of the dimensions of slowness you haven't mentioned is that if I'm a validator in Chile and I'm attempting to collect the entire surface of RPKI, I have to have a reliable path to Indonesia and to Mongolia and to Europe and North America and my buddies nearby in Uruguay. I need to be able to get to all of them to complete my act. If this was in an intermediary, and the CDN had an anycast surface, I would be fetching at a constant distance cost to that CDN with a better protocol. Is this a potential? Job Snijders 33:36 Yeah, yeah. So because we're fetching by hash, not by file name, not by FQDN. You can connect to anycasted instances and rotate between the instances. So we've now converted the RPKI distribution problem into something that is George Michaelson 33:51 this is now similar to a model like 1.1 dot 1.1 Job Snijders 33:56 and 8.8, dot 8.8, and nine, dot 9.9, and whatever open resolver you want to use, we have now constructed a system where you can fetch RPKI data from publicly accessible endpoints that contain data for multiple RPKI origins. So as you point out like this, this latency factor from the client to all over the world, the latency becomes a predictable elements. You can fetch from a close by mirror, and you can use multiple mirrors concurrently. So if one instance is a little bit behind or outdated for one reason or another, it doesn't matter the client recognizes, oh, you contain old data. Well, I'm going to try the next Erik relay. George Michaelson 34:40 So if I consider myself in the role of a relay that front facing service through Erik, I have the potential to use either Erik or RRDP or rsync in my back, depending on capabilities in the real publication point, don't I? Job Snijders 34:56 So an Erik relay merges all the data be it received via rsync over fee four or IPV6, received through RDP, received through Erik, it synthesizes a final construction of the data set, presents that to the clients in the Erik format. But it doesn't have a monopoly on the data. The client is absolutely free to connect to multiple relays operated by multiple organizations. So I think this is a huge improvement in robustness. George Michaelson 35:28 Oh, it sounds like a very positive outcome. You were presenting this in SIDROPS this week? Job Snijders 35:33 Yeah, I presented updates, and one of the fun things to share in that update is that there is running code. So I have set up a number of relays. Some of them are, you know, only in Europe. Some of them are globally distributed. So I created a diverse set of relays to help test what does the performance of Erik look like from a global perspective? And Tom Harrison from APNIC did a wonderful job creating a Erik client implementation. So now we have all the building blocks to do extensive measurements over the span of multiple months to with high precision, observe what is the difference. If there is any difference between Erik, RRDP and rsync, and my hope is that the Erik testing instances will show to consume less data, be faster in converging but we'll see. I mean, George Michaelson 36:27 well, we'll have you back and talk about this another time when this emerges. Job, that's been absolutely fascinating. We'll put pointers to this into the blog that goes with the podcast. Thank you. Job Snijders 36:38 And if I may, do one more call out, if you're interested in researching these phenomenas, we are developing the Erik protocol at this moment. So for students that are interested in this type of problem space, like, how do you synchronize hundreds and 1000s of tiny objects to 1000s of Computer George Michaelson 36:58 There are Research Opportunities Job Snijders 36:59 Exactly? And we need the help from younger students to progress this protocol to be something that we want to use the next 10 20, 30 years. George Michaelson 37:09 Fantastic. That's great Job -thank you. Job Snijders 37:11 All right, George, until next time, George Michaelson 37:14 if you've got a story or research to share here on ping, why not get in contact by email to ping@apnic.net or via the APNIC social media channels, also remember the measurement@apnic.net mail list on orbit is there to discuss and share relevant collaborative opportunities, grants and funding, opportunities, jobs and graduate placings, or To seek feedback from the community on your own measurement projects, be sure to check out the APNIC website for All your resource and community needs until next time you you.