Beau Gieskens 0:00 Yeah, I guess part of the link with the measurement forum for us is that we're actually using a lot of the measurement data from other sources. So one particular example that I was kind of discussing during the talk at the measurement forum is with like writes, writing Information Service or route views, where they have these BGP collectors there, in a sense, measuring what kind of BGP data is coming across the Internet, and we are a direct consumer of that measurement information, and we use it to see what is the BGP state up there, and we use it as the basis for a lot of our routing status calculations in Dash. George Michaelson 0:46 you're listening to ping, a podcast by APNIC discussing all things related to measuring the Internet. I'm your host, George Michaelson, this time, we're continuing our series of small recordings made at the recent PIMF session from the apricot meeting held in Petaling Jia in Malaysia last month. PIMF, or pulse Internet measurement forum, is a new community initiative launched by Amrish Phokeer and Robbie Mitchell from the Internet Society. ISOC. Want to help grow a community of people who are interested in Internet measurement in the widest sense. This episode of ping wraps up our interviews held at the forum. This time we're going to hear from Beau Gieskens from APNIC, Doug Madory from kentik, and Lia Hestina from the RIPE NCC. Before this to reset context, I'm replaying the recording we heard last time about PIMF from Amrish, who helped establish the meeting. Hello everyone. Welcome to a slightly different PING this time, instead of having one guest and talking at length about a measurement activity, we've put together a package with a number of short recordings of people who attended the first pulse Internet measurement forum. PIMF. So to kick things off, I've got Amrish with me here, who has helped start this new cycle of meeting. Amrish, welcome back to ping. I think the last time we met, we were in Vietnam, weren't we? Amreesh Phokeer 2:13 Hey, George, yes, we were in Vietnam, and I remember we had this very nice meeting under the mango tree, and then in the background we had the train passing. [George: That's right], the recording, George Michaelson 2:23 well, it's a little different this time. We're in Petaling Jia near Kuala Lumpur at the apricot meeting, and you have established a new activity that's meeting at other venues called the pulse Internet measurement forum, or PIMF. Can you explain to people about this, Amreesh Phokeer 2:41 yes, so for those who don't know, at the Internet Society, we have the project called pulse. So pulse really is a product that produce Internet health metrics. So really, to give people an idea of how good or how bad their Internet health is in their respective countries. George Michaelson 2:59 Now we spoke about this in Vietnam, and it's in the archive of ping recordings. But to refresh most people's idea of Internet measurement is that it's very focused on technology behavior, packet loss, delay, BGP problems like this or like that. But pulse is actually quite a lot wider than that, isn't it? It's a measure across a lot of different areas. Amreesh Phokeer 3:21 It is, it is, and we want to make it more holistic. And also we want to put data in there that is also relevant to other people than the technical community, because at the end of the day, the Internet is for everyone, and this is what we preach at. The Internet Society and every member of the community, whether they are regulator, a policy maker, need to find some information relevant about the Internet and hopefully be able to produce some actionable items to improve the Internet over time in their country. George Michaelson 3:53 So there's a community of people who are interested in measurement in the broadest sense, in the Internet. And the Forum is an initiative you guys have taken to sort of bring that together into some kind of collected state, isn't it? Amreesh Phokeer 4:07 Totally so we found out that our community is very varied. Again, it can consist of very technical people, but also less technical people, especially those working in the information control space where they are fighting against things such as Internet censorship and Internet shut down and they don't want to have relevant information on for example, what are the techniques that governments are using to shut down the internet? And what can, for example, what can they do to protect their citizens against, for example, surveillance? So these are things that we want to talk about in the Internet measurement forum, and usually those people wouldn't have a place or a forum to share ideas, I would say, perhaps in a secure and safe environment. And this is what we want to provide on top of also providing insights about the Internet of how things are. Solving. We also want to give the opportunity to the different members of our community to also bring us their own way of thinking, the ideas of how to improve the internet. And this is what we want the PIMF to be a really an interactive place where people would share ideas and also learn from the whole world of Internet data, which is out there and not necessarily very technical, always technical. George Michaelson 5:27 So we heard yesterday from Stephen song, who's based in Halifax, Canada, and he's been working on mapping techniques. Quite a lot of people are aware of the dependency on international transoceanic fiber. But Steven has been looking, for instance, at the domestic, terrestrial fiber maps and the lack of consistency and information gathering over what fiber, what fixed line and broad band services are actually laid down as wires and assets in the ground, and at the structural issues around people coming to government or to civil society and saying, I guarantee you an independent path, but in reality, they might actually be using the same bearer. So that's kind of one side. And then we had other people who were presenting broad Internet measurement platforms like Atlas. We had a talk from Beau Gieskens at APNIC about some techniques they're using to do alerting. It's a nice mix of people that are coming into the room, isn't it? Amreesh Phokeer 6:23 Yeah, it's a nice mix of people. And you want to keep it that way. We want to increase the interaction between the technical people and the non technical people, because we want, eventually, the non technical people to be able to use this information and to design policies that eventually would make the Internet a better piece. George Michaelson 6:42 Meetings are really quite expensive to come to, and in the Asia Pacific, we do have a lot of emerging Internet economies where it's a marginal problem. It's quite a big issue for people to invest in coming to what is essentially a fairly small meeting set against all the other things. So is PIMF going to continue to co locate with other activities for the foreseeable future, Amreesh Phokeer 7:04 not necessarily. We are also planning to do one of events, and that's usually based on the needs of the community. So for example, recently we got an invitation from India. So we usually work with our chapters around the world. So there is a chapter in India, the Kolkata chapter, who are very much invested in the Internet measurements. George Michaelson 7:25 One of the people who was presenting here is from a community who have built the Indian indigenous platform for Internet measurement. I think we might be talking to them later on. So you're kind of saying that if you are in the South Asian community, PIMF, might be coming into your space, into a place where it's easier for you to come and participate, it's going to orbit around a bit. Amreesh Phokeer 7:47 It's going to orbit around a bit. And we want to make it a very localized event where we talk about the local issues and the local challenges, and how eventually, putting all the data together, we can help the community make better decisions. George Michaelson 8:01 So this is a workshop style meeting. You don't yet have a program committee. It's really come as you are and have information and stories to share. Is that right Amreesh Phokeer 8:11 for now? Yeah, as we just started, so we are, I guess, gathering momentum a little bit so really understanding what people want and what people would like to hear, but eventually, yes, we want to make this a more structured workshop or forum, so we would, as you mentioned, probably have a program committee around it, and also let people tell us what they really want to learn about. George Michaelson 8:35 Is the way to participate, to actually be in the room on the day pimp is meeting. Or is there another way people can engage in this work? Amreesh Phokeer 8:42 Well, we have broadcast the live stream, and people could also connect over zoom. But maybe in the future, we will make this a more structured way where people How can also interact. George Michaelson 8:52 And you're thinking about having an archive of material is there's going to be a web page associated materials. Amreesh Phokeer 8:58 It's already on post, the post platform, George Michaelson 9:01 and this is a continuing initiative of the Internet Society. So pulse is an activity of the Internet Society, and PIMF is going to be an activity about pulse in the context of pulse of the Internet Society, but in association with other groups as well. Amreesh Phokeer 9:17 Yeah. So both, as you know, is a project of the Internet Society. And within the project of the instance of the Internet Society, we are running this activity, which is our outreach activity, disseminating the information that we are collecting. Because usually we might be working on interesting projects about different aspects of the internet, but usually it gets buried into the loads of information that we have on the pulse website. So the idea is actually to use the research that we are doing and then try to see if this is of any benefit to the community. George Michaelson 9:48 So is there a place people can go to learn more about this meeting cycle? Amreesh Phokeer 9:52 People can go on https://pulse.Internetsociety.org, it's also a newsletter that people can subscribe to, and they will have information. And on a monthly basis about the different things happening. George Michaelson 10:02 Excellent. Thank you, Amrish, thank you, George. Let's hear now from Beau Gieskens APNIC information products. Beau is a senior software engineer at APNIC and has been working on some structural improvements to the DASH system. DASH is the dashboard for network health, a member service which can show you information about your own network assets. It tracks bogons Your network may be emitting in BGP, your MANRS, readiness score, and it has information about your origin AS and visibility of addresses in the APNIC Honeynet system, showing the hits which originate from inside your network. DASH has a system for issuing alerts, and this had some severe scaling issues. Hey, Beau welcome to PING. Can you tell everyone who you are, please? Beau Gieskens 10:52 Yeah. Hi George. I'm Beau. Gieskens. I'm working as a senior software engineer at APNIC. George Michaelson 10:57 So you came to the ISOC PULSE meeting and talked about DASH. Can you just explain to people what DASH is, please? Beau Gieskens 11:05 Yeah, sure. So DASH is kind of APNIC network health dashboard that we make available to our members at dash eight.apnic.net, and their members can do all sorts of things like create alerts to be notified when they have some kind of routing security thing going on, or if they have suspicious traffic originating from their network, it's a place where they can view the MANRS, readiness score for their networks, all sorts of things like that. And we're always expanding with new things. George Michaelson 11:35 So the pulse forum was a collection of people who are interested in measurement, and it sounds like DASH has quite a lot of measurement features inside it. It's a diagnostic tool, but it's also something that can tell you about the current state of your system. Beau Gieskens 11:49 Yeah, I guess part of the link with the measurement forum for us is that we're actually using a lot of the measurement data from other sources. So one particular example that I was kind of discussing during the talk at the measurement forum is with like RIPE's routing Information Service or route views, where they have these BGP collectors there, in a sense, measuring what kind of BGP data is coming across the internet. And we are a direct consumer of that measurement information, and we use it to see what is the BGP state out there, and we use it as the basis for a lot of our routing status calculations in DASH. George Michaelson 12:25 Can you tell us a little bit? Can you give us, like, the short version of what was the story you brought to the pulse meeting Beau Gieskens 12:31 In DASH, when we were calculating routing status, initially, we would essentially say, Okay, I have these prefixes or these ASNs that I care about where I want to know if there's a routing status issue. And so I take all of those, and if every single one of them, I'd have to launch out a network request to a series of services which we want information from. So that could be IRRD and routinator, and in the case of BGP, we had an internal service which was kind of modeling the state of one of the BGP collectors. But anyway, we're querying all these services a lot of times. George Michaelson 13:07 Well, if you're running a service for registered members who said, I want to be alerted of change in my system, it means every hour on the hour in the cron job or something, you're having to go and do these queries repeatedly Beau Gieskens 13:20 exactly. And this is what was happening. The thing is, it worked well. It worked well when there weren't that many alerts, or if everyone who had created an alert only cared about one prefix. But the fact is that people have bigger networks than that, George Michaelson 13:31 really quite significant, large numbers of prefixes. Each one has to be tested individually. Beau Gieskens 13:36 Yes, and a lot of these services didn't support querying a whole range of things at the same time, or anything like that, and because all of them are separate, and we have to query kind of a large amount of data and then refine it down to what we actually care about for notifying members about a routing status issue, it was a bit of waste. George Michaelson 13:54 It's as if I want to be told if something goes wrong in any part of my routing system, and you get to make one alert that says, tell George when something's wrong in his routing surface, but it involves asking about every prefix George has. And if I've got 100 prefixes, you have to construct 100 queries into the information spaces to determine if one alert message should be sent. Beau Gieskens 14:18 Yeah, that sums it up. Very well. George Michaelson 14:20 What did you do to get over a scaling problem here, we Beau Gieskens 14:23 kind of flipped the problem on its head. Rather than saying, Okay, we want to look at all of George's resources and see each of them. What the problems are. We just looked at all of the information that is coming out of these services. So in the case of IRRD, you have an event stream where you can get every single event where something is created or deleted. And so for rodinator, they have a JSON Delta end point you can query where to give you the entire state. And then after that, every time something is created or deleted, and we kind of created our own version of that for. Internal BGP service. And so we would take all that information and just watch the events that come through. So this is an example, if you get a new IRR route object that comes through, I'm watching that event, and so I'll check it against my filters from all of the alert and it's like, do I care about that event that just happened? If so I know which alerts that it's relating to, and I can do the rest of the processing from there. And it probably sounds a bit complicated, and I find it's difficult to explain to people without showing pictures of it, George Michaelson 14:23 but the way I think about this, events are like a stream. They really are like a continuous stream of things moving past. And it sounds like you made the decision to invest in bootstrapping the initial state of everything. So you do one big thing at the start of running this system, and then you watch the stream flow by. And instead of having to consume the whole of the stream and hold all of the stream, you're basically combing out the bits floating in it that are relevant to whatever you have in any of your filters that are the alerts that you're constructing, aren't they? Beau Gieskens 16:04 Yeah, that's pretty much true. George Michaelson 16:05 And so this changed the cost basis of running the service, Beau Gieskens 16:09 yes, although maybe not in the way you might think. So there was a bit of an upfront investment in the cost to run these services, which are watching all the sources. So now we're running seven BGP adapters to watch all of that information, and some of those mean quite a lot of memory to track all of the George Michaelson 16:28 so it's the classic computing trade off, that you can achieve an outcome that works in a timely manner, but you have to invest in the cost of either computing burden or memory burden. There's no path out. It's kind of like CAP theorem, isn't it? Beau Gieskens 16:41 Yeah, I guess the point is that we pay a bit more up front, but it means that for the end user, they have a system that works more reliably and actually notifies them. George Michaelson 16:52 I think I heard you say when you started this, you were able to run your cron cycle on a really quite tight boundary. You could kind of run almost every couple of minutes, or every five minutes. But when you'd grow into a certain size, it was taking more than five minutes to run one complete iteration of the cycle. So you'd hit a real scaling barrier in the timing limit there, hadn't you? Beau Gieskens 17:14 That's right, and it was kind of bad at one point where it was like, Oh, the thing just processing the Alerts is running out of memory because it's doing all these things, or, Oh, it's actually launching too many requests to the service, so we have to scale something else up, or lots of weird issues like this. It wasn't even just the fact that we needed to process some alerts every one or two hours because they were too big. It was kind of burning us in every possible way. George Michaelson 17:40 So after you implemented this event sourcing based model, what kind of time boundary did you get back that you were able to run the alert cycle on? Beau Gieskens 17:49 Well, the thing is, we actually don't run an alert cycle anymore. George Michaelson 17:53 Oh, there's no cycle, Beau Gieskens 17:54 right? So for anytime someone is getting notified now in Dash, it's pretty much just because we got an event through in the stream. George Michaelson 18:02 So it's triggered by seeing the event happening at the rate you see events occurring on the input side. Beau Gieskens 18:10 That's right. So I think the latency for that is probably within five seconds, if we see it in RPKI through route meter, then within five seconds, the user should be getting notified. George Michaelson 18:21 Oh, that's a really nice outcome. That's a good deliverable. And you had to build a number of different, I think you called them adapters, to convert the event streams you were seeing into a common representation. Beau Gieskens 18:34 Yes, that's right. So I mean, while we're doing everything event based from these sources, each one of the adapters is also writing what it knows to a shared database, because we still do need to query things on demand. So if someone has just created a new alert, and we can't just sit there and wait for the events to come through, so we still have, like, a database version of all the things we care about, and we can query that say, like all these prefixes, what's the routing status like? And that's like a pre prepared, already formatted about everything we care about, view of all the rating status stuff, and it's miles faster. George Michaelson 19:12 So Beau, if I understand this right, you're now able to be responsive to an input event that might be only five seconds old. But doesn't that present a potential problem if you have a flood of events coming in? Beau Gieskens 19:25 Well, yes, and we did see that, and we've kind of implemented our own thing for handling that. So what we do is, when that first event comes in, we'll notify the user immediately so they have, like an inkling that, okay, something is going on, I need to start looking into it. We have like, a five minute cool down from there. So any new events that come in, we're kind of batching them together. And then after that five minute window has passed, they'll get, like, a more complete notification about every all of this has now happened in the past five minutes. George Michaelson 19:59 So. A bit of hysteresis and a bit of buffering, and you don't then create a storm of notification events. If I remember correctly, DASH is actually multi modal for output. You can tell people in different ways. They can choose how they're alerted. Beau Gieskens 20:14 That's right. So if you are worried about maybe I haven't seen something yet because I'm still waiting on that cool down, you can always go to the DASH web interface, and you can see the exact current status of the alert at any time. George Michaelson 20:27 Thanks, Bo, that's really interesting, and I think it's lovely to see that there's a community outcome of having people providing services like RIS and route views, and then being in a forum like pulse to talk about both the supply side and the consumer side of those things to produce a measurement outcome. Beau Gieskens 20:45 Thanks, George George Michaelson 20:46 Thank you. Next, we've got Doug mature from kentic. Doug spoke on ping last year about RPKI and BGP. We'll get a refresh for the current year, another time on ping. But in the meantime, I spoke to Doug after his talk on a quirk of BGP called AS sets, which has been responsible for some very large route leaks. Doug, it's great to catch up with you again. Doug Madory 21:10 It's great to be here. George Michaelson 21:10 So here we are at apricot at the first PIMF, the pulse internet measurement forum. Now, usually when we've had you on ping, you've been talking about RPKI, but for this meeting, you actually brought a different story to the table, didn't you? Doug Madory 21:25 Yeah, that's right. So before we had ROV, our most recent routing security technology, there's some older techniques that are still in use and are widely used today to build filters to try to block inappropriate routes from coming through BGP sessions, and some of those are built by pulling in data from IRR databases. George Michaelson 21:45 So IRR Internet routing registry, this is the kind of structural formalism in people who are used to who is in the domain space. It might look like a structured record, but it's actually up to the local registry for that domain space to decide what form. It has few rules, but pretty much it's free form. And ARINs record of note for asset management, it uses its own structural form. That is a type value notation IRR. Now they are built using the notation the RIPE NCC built, r, p, s, l, routing policy specification language, is that right? Doug Madory 22:22 That sounds right to me, something you know, a better idea. George Michaelson 22:24 So you were looking into this space of RPSL objects that are defined in the Whois form, and they used to construct BGP filters, [Doug: correct]. What's the idea here? Doug Madory 22:35 One of the techniques is to use a construct called as sets. And so unfortunately, in BGP parlance, we have multiple terms that are overloaded. So have multiple George Michaelson 22:46 rights. So there are two meanings minimum of as set explain both, Doug Madory 22:50 yeah. So there is an AS set. This as_set refers to this route aggregation in BGP that tries to merge announcements for I'm sure, the motivation behind it, if you see it in the wild, you see, like the little curly braces in the as paths, George Michaelson 23:03 the notation that's typically used in the standard view of textual representation of a BGP route. How's that? Doug Madory 23:13 One of the problems with that is it also breaks the association between which prefix was originated by what AES. You can't use the AES path right the origin. George Michaelson 23:21 So that's the one in BGP view, yeah, one that you would look Doug Madory 23:24 That's right, that's not the one I was looking at. The other as set. I was looking at the as set. That is a convention that you use to try to aggregate ASs for routing policy based on IRR database information. George Michaelson 23:36 So it's when I as a authorized person in who is in the IRR data we're talking about. I make a declaration. Here are a set of ASs. I want to refer to by name, [Doug: right] And you can use this in many different ways. I mean, the typical way is that you're a BGP speaker, speaking to customers who also speak BGP. And you want to construct a set of things that for your purposes, you call George's customers. Now, they don't get to decide that they're included in this set. Do they? It's my decision. [Doug: Yes]. And I could also have a set that was not about customer facing behaviors. It could be about my immediate peer adjacencies outside my boundary, and I could call them George's peers, and again, they don't have a right to say if they're in that list or not, [George: right]. But I'm building these filters. I'm building this information for the rest of the world. And the rest of the world has to know George is going to talk about George's customers and George's peers. That's kind of strange. Doug Madory 24:40 What you're describing is a lot of flexibility with this, but also with a lot of flexibility comes the risk of data quality, and that's kind of what I was getting at at the talk this week, right? George Michaelson 24:50 So you've been exploring the problems in this space, Doug Madory 24:52 yeah? So I opened with a routing leak that took place, and there's lots to choose from of one that, yeah, that involved a. The perpetrator had an as set. Who, if you were to expand the as set, it included 43,000 AS's, George Michaelson 25:06 sorry, how many? Doug Madory 25:07 43,000 ASs George Michaelson 25:10 that's half the total surface of the default free zone, Doug Madory 25:14 yeah, that's like, so it's too big. It's not just too big. This causes lots of problems, and it's not the only one. So, you know, with an allow list that's built from an as set that includes 43,000 plus ASs, you've kind of white listed, or allow listed, you know, just about the whole internet, or large portion of it, defeating the purpose of trying to, you know, enumerate what are the things that you're supposed to be passing. George Michaelson 25:38 So this is quite a common situation when you think about mistakes made in configuration systems, in a way, as set is kind of like the old idea of a macro. It's a condensed way of referring to a really large, complicated set of things with one term. And the problem is, somebody out there built an as set that had half the global Internet in it, and they made an assertion in IRR about that, not really intending to refer to the global network, but for their own internal purposes. Thought they were saying something useful, but I'm guessing it didn't have the effect they thought it had. Doug Madory 26:15 Yeah. I mean, I don't know the psychology behind how it came to be, but I think what ends up happening is you can include other as sets in your as set. George Michaelson 26:23 You can nest them? Doug Madory 26:24 Yeah, you can nest them. They can be recursive. You know, you can include an as set that includes your as set in your as set. So you can imagine it just, I think, very quickly, people just lose track of what is in the as set, and it just grows and grows. And I think, you know, there's a motivation that you would like to probably avoid blocking not including something that you might actually use. So by constantly erroring on a side of caution and including more than restricting, these things just kind of grow unbounded. George Michaelson 26:52 So kentic was looking at this. How did you uncover this state? What were you looking at to do this? Doug Madory 26:57 I guess I've known about this issue for a while, and I think when I saw this leak pop up, and I check in several I'll dig into leaks happen. You know, for everybody who doesn't live in BGP space, they're happening still fairly regularly, but the state of routing hygiene has gotten to a point where they're really not that disruptive, which is good. This is a success. George Michaelson 27:16 So route leaks are when people announce things that aren't necessarily truly about what they do. But because they're announced into BGP, the rest of the world acquires knowledge of things that they say they can do, and there's potential for these paths to actually become active, visible paths other people believe they should be using. Doug Madory 27:36 Yes, it's a good way to say, yeah, it's a routing mishap, something like and it usually either ends up misdirecting traffic or disrupting George Michaelson 27:42 and the filters, to some extent, are an attempt to say, you know, if I'm listening to you, the only things I'm going to believe from you are the things that I know you said were your customer space. And that way, the leaks that might happen from you, I kind of occlude. But this as set mechanism that would be a way for you to say Doug's customers if you misuse it, and if it recursively applies to something else that includes someone else's statement, I could acquire massive amounts of knowledge of belief that you do things, and I would white list them, and the filter doesn't do anything useful. Doug Madory 28:17 Yeah, it's even worse than that. Aside from it not doing anything useful. There's an expense to it. So this is a lot of data. These things get, typically, are we? The only way we're going to do tackle any of these problems is through automation. And the IRR database, for all its faults, enables automation. You can just pull this data down. There's nobody manually making this stuff. You can just programmatically create these, apply these to interfaces. And so we need the system to enable the automation, but there's a cost now you're pulling down, you know, an enormous amount of data, all of which is going to be useless. You expand this into a router configuration that's over a million lines long. In this particular case, if you were to actually apply that to your router configuration, there's a cost to your router. You're doing this at least once a day, and it's all just defeating the whole mechanism. George Michaelson 29:02 Yep. So it's a classic tragedy of the common. There's one mistake by somebody doing a configuration design means every BGP speaker consuming this data has an explosion of data, cost and of complexity in their configuration, and potentially actually killing their router trying to load the configuration. Doug Madory 29:20 Yeah. So I've talked to some cloud operators and large carriers of who asked, Do you guys still do this? And like, yeah, we have to. We have to, but you have to build work arounds for these cases. Because we met, we talked about one incident from last year, but there's actually 1000s and 1000s of as sets that expand to ridiculous amounts of ASs. So they have to come up with ways to survive. George Michaelson 29:43 So the other kind of as set, there's been a standards process pushing back really the other kind the BGP as set, we're talking about deprecating it is kind of not really the way to do it, right, but this as set is probably a useful tool if we could. Put it back in its box. Doug Madory 30:01 I thinki t's going to be with us for a while. We're going to depend on it. I mean, so this is very helpful in the like a adjac leak or a path leak thing, where you have a leak, you're a routing mishap, but the perpetrator isn't originating the routes where ROV might be able to help. ROV can't help in the path leak, and like an incident, the MTS incident from last year. The origins are intact and the routes, that's not the problem. It's a problem. It's happening in the middle of a s path. So, you know, the things that we're trying to do to address that are the ASPA technology, that's that's not out yet, but we're, yeah, this as set IRR database, that we're going to need that for the time being. And so then I guess my motivation and just bringing this to like. So, like I said, I've I encounter this in my research. I also hear about it. I hear people talking about it, and I never seen anybody kind of like, dig into this, and it seemed like it'd be there'd be something positive to just highlight. Like, hey, if you are a network and you have an as set, can you just make sure the contents that are in there, you're being as parsimonious as possible, and maybe even avoiding putting nesting another as set inside, because I think I'm guessing that's where people start to lose track of what all is in the as set. George Michaelson 31:09 It sounds like the perfect kind of story to bring to something like the PIM meeting other people who are sitting in the room might have ideas around this, tools they could build, ways they could look at this space, look at the problem. It's a nice thing to talk about. Doug Madory 31:22 Great, yeah, I thought so. George Michaelson 31:23 Thanks, Doug, Doug Madory 31:24 you're welcome. George Michaelson 31:25 Lastly, let's listen to Lia hestina from RIPE NCC. Lia is the community development officer focusing on the ripe tools deployment for the Asia Pacific and Africa. She is the lead in our region for Atlas, the RIPE NCC measurement system, and has been working on the project since its very beginning. Lia, how lovely to see you again. Lia Hestina 31:49 Lovely to see you too. George, George Michaelson 31:50 So here we are at the apricot meeting, and you have just been presenting in PIMF, the pulse Internet measurement forum. [Lia: Yes]. Can you tell people a little bit about who you are and what you do? Lia Hestina 32:05 Well, yeah, my name is Liaand I've been with RIPE NCC quite a while now, since 2009 George Michaelson 32:13 Oh, Veteran of the industry! Lia Hestina 32:17 yeah. So actually, I started at the front office at RIPE NCC, and I keep on moving to different department, learning new things, and ended up as part of RIPE Atlas team that I really enjoyed and knew back then in the Internet industry. So yeah, I really enjoyed all the time that I've been with the RIPE NCC. George Michaelson 32:38 So Atlas is something we know a little bit about in our community, but for people who may not know about Atlas, can you describe this project a bit? Lia Hestina 32:46 Yeah, of course. So RIPE Atlas is an Internet measurement platform. What makes RIPE Atlas special is we have many probes connected in different networks all over the world. We have around 12,000 probes connected right now. And what I love about it, what makes me believe in the project, is that it's giving access to everyone to this data so that they can measure their network, or any network that they want to monitor using all this vantage points that we have around the world. George Michaelson 33:19 So the probe is something that ripe developed itself, isn't it? [Lia: Yes, indeed] it's a small device. I have a very old one, and it's kind of the size of a box of matches. I think the newer ones are a little bigger, but older or newer, little or large, they all fundamentally do the same thing, don't they? Lia Hestina 33:38 Yes, indeed, they are more or less the same thing. So I think right now, we are at the version five, [George: five], yes, and we also have the software version to make it easier for everyone, you know, in network operators, especially, they don't have to wait for the hardware to be available. Because that's also one of the thing that has been a challenge for us. You know the availability of the hardware, that's why we are now at version five. George Michaelson 34:05 Yeah, so the probe isn't going to take over your network and absolutely flood it with packets. Is it? This is a quite carefully designed system for performing measurements. Can you talk a little bit about how the probes measure things? Lia Hestina 34:19 So when you host a probe, when you have this tiny device, when you plug it in into your network, the probe will start connecting to our servers. It will start measuring all the root name servers, and we show the result in built in measurement graphs. And yeah, so all of these things is available, already online, George Michaelson 34:40 And it's really just a small amount of traffic that's being sent in the background, but you have this second layer where people who are in the system and who have registered a probe or have registered an interest can use credits in your system to perform their own experiments. Am I right? Lia Hestina 34:57 Yes, and that's what makes rapid loss also spend. Show we have the credit system to make it fair for everyone, so that everyone has the same access. And how you can earn credit is by hosting a probe. So and it's also to protect the system, right? We don't want to overload the system. So, yeah, when you host a probe, one probe, for example, you will get around 21,000 credits on a daily basis. And with those credits, you can run your own measurement. George Michaelson 35:25 And this is not only to go and probe the root surface. You can probe or ping or trace root or HTTP connect to a number of places. Can't you? Lia Hestina 35:35 Yes, there are six type of measurement that you can run with rapid loss, ping, trace, Route, DNS, SSL, TLS and for HTTP measurement, we only limited up to four anchors. So you can use anchors to run HTTP, George Michaelson 35:52 Anchors are slightly different from probes? Lia Hestina 35:53 Yes. So anchors is like we like to call it RIPE Atlas on steroids. So RIPE Atlas is a little bit different than the ordinary probes. It attracts measurement from all the anchors that we have connected, so we have around 800 George Michaelson 36:09 so this is like a full mesh where the anchors talk to each other, but they also receive measurement requests from the tiny Atlas probes. Atlas probes themselves aren't the targets, and if you have one running in your home network, you're not going to get denial of service attacks into you. You're emitting requests out into the world. [Lia: Yeah] so hosting an anchor is a good thing to do, but it's something you need to think carefully about the traffic implications, and you say you have 800 of them now. [Lia: Yes] that's huge, yeah. So you have 10 to 12,000 probes and 800 anchors continuously measuring the network. That's amazing. Lia Hestina 36:52 Yeah, I think that's what makes people interested in rapid loss anchors, because with rapid loss anchors, you don't have to do anything, right? You are measured from all this other George Michaelson 37:02 continuous flow, the behavior of your part of the network, Lia Hestina 37:06 and you will learn about how reachable your network is from all these anchors. George Michaelson 37:12 Yeah, so in the talk that you gave, actually, I think it was Alistair Strachan from RIPE who presented, initially the ripe system, as it was described. You showed the map of the world, and you have great density of probes in continental USA and all through Europe and in the Middle East. Asia is a little thinner, isn't it? Lia Hestina 37:36 Yes, it is. And that's also the reason why I go to apricot meetings, to meet the community. And you know, just being here, it's much easier to interact with RIPE atlas ambassadors. We have many RIPE Atlas ambassadors in Asia Pacific, [George: yeah]. But there are still many places, many networks that are still uncovered, that we still want to cover with RIPE Atlas. George Michaelson 37:58 So it would actually help a lot if we could get people to get this system running in our footprint in the Asia Pacific region, this could help boost measurement coming from this part of the world. [Lia: Definitely, yeah] But I think, as Alistair said, it's not enough to get one. You've got to plug it in. Lia Hestina 38:15 Yes, yes. That's one of the challenges. Of course. You know, sometimes we meet people. So we have this rapid loss Ambassador running around in meetings in NOGs, distributing groups. Often people are very excited to hear RIPE atlas for the first time, they got group, bringing them home. And then once you're home, there are many responsibilities that you have to do. You go back and goes on the shelf, and that's one of the challenges that we face. But oftentimes they go immediately online. And, you know, like I said yesterday, I met this community from the Philippine I thought, like, they're amazing, you know, the support, the willingness to, you know, let's do this now. Let's do this today. [George: Yeah], and you get this, George Michaelson 39:00 it's good energy isn't it Lia Hestina 39:01 Good Energy and making you feel like, Oh, it's good to be here. Yeah, you know. George Michaelson 39:07 And the pulse Internet measurement Forum is a good place for you to engage. I got the feeling quite a lot of measurements being done in the community. Actually, leverage Atlas is part of the work they are doing. You're like an underpinning for other people to do experiments, aren't you? Lia Hestina 39:23 Yes, we work a lot with ISOC pulse. There's a project which is called Mira that Amreesh was presenting on Monday, and that project is using rip at last data and MLabs, so they're using this pod, the Mira pod, in collecting this data using wrap Atlas software probe, George Michaelson 39:45 yes, I hope to be talking with them later on as well. Lia, it's been absolutely wonderful. Thank you for talking to us. Lia Hestina 39:51 Thank you so much for having me, George. George Michaelson 39:53 And if people want to read more about the Atlas Project, the URL they go to would be Lia Hestina 39:58 it's atlas.ripe.net George Michaelson 40:01 Great. Thank you. Lia Hestina 40:03 Thank you so much. George. George Michaelson 40:05 Well, that's all from this print forum, but we'll do more of these when it happens again in our region. If you've got a story or research to share here on ping, why not get in contact by email to ping@apnic.net or via the APNIC social media channels. Also remember the measurement@apnic.net mailing list on orbit is there to discuss and share relevant collaborative opportunities, grants and funding opportunities, jobs and graduate placings, or to seek feedback from the community on your own measurement projects. Be sure to check out the APNIC website for all your resource and community needs until next time you.