Geoff Huston 0:00 Should a BCP describe what we should be doing or what is available to do? And it's this kind of thing of, is a BCP a shopping list, or is it a wish list? You know, are we writing to Santa Claus, or are we heading down to the local supermarket? If I want to put it in more mundane terms, because the latter, the supermarket, it's real. It's tangible. It's on the shelves. It exists. And if a BCP says, well, tick this box and not that box, then I can do that, whereas, if I'm writing to the Easter Bunny, the answer really is, hey, rocket ship to Pluto. Off we go. You know. George Michaelson 0:35 you're listening to PING, a podcast by APNIC discussing all things related to measuring the Internet. I'm your host, George Michaelson, this time I'm talking to Geoff Huston from APNIC labs again in his regular monthly spot on PING. However, this time, we're having a slightly different kind of conversation, one that was caused by a side comment in the last episode we recorded about the DNS. It comes from the idea of a best current practice or BCP document. Geoff and I feeling a little bit like dinosaurs watching more agile mammals take over. We seem to share an older, quite different idea of what a BCP document is. We think it should stand for the best of what is seen right now in protocols driving the competitive market of systems built against standards. We see a trend emerging to treat a BCP as more like a wish list of things we'd like to do in the best of all possible worlds. Aspirations for a better place to stand are really a very fine thing. I don't want to think either of us would be seen as opposing that kind of advocacy. But when it comes to documenting what is the observed current best practices, we think that they have to be grounded in what is, not in what could be. Geoff, welcome back to PING. What shall we talk about this time? Geoff Huston 2:17 Oh, hi, George. Hi, hello, listeners. I'm kind of miffed George, my nose is subtly out of joint, so it's good that we're not using video here. [George: oooh]. So what I'd like to talk about today is something that's caused me to, I suppose, have some vague tremors of disquiet in the world of the IETF and the standards process. George Michaelson 2:36 Let's see where this goes. Pray continue. We've just recorded this piece discussing aspects of measurement and the behavior of glueless DNS, and it was a very nice technical conversation about DNS behavior and observations in the experimental space, but you highlighted something that I think is more it's more interesting and it's more potentially risky and intractable, but I really am fascinated by it, and I think we should talk about it here while we have the chance. So Geoff, this BCP thing, something smells wrong here Geoff Huston 3:11 Right. The entire process of the IETF was summarized by Dave Clark. I think he's still associated with MIT. He's getting on a bit, but still, his brain still working. He's a fine, a fine dude, and it was summarized by the code, by the the mantra, rough consensus and running code. The whole idea was that the documents produced by the IETF were meant to be of such a quality that they were able to produce implementations in code, and those implementations would interoperate. [George: Yeah]. So if you read an RFC and you went to the market going, I want this, you know, your product to conform to RFC, 111, then, in essence, everything in RFC, 111, should have been implemented in that code. George Michaelson 3:58 And the running part wasn't really a projection into the future of when we've invented time machines, we'll have this. It was we really honestly see this as a viable platform and are expecting or have functioning code. Geoff Huston 4:14 Well, there are RFCs and RFCs, and if you look at it, you start your journey as a proposed standard, and that's rough consensus. We kind of agree that the code that will be produced will do the following, and that's an aspirational document, [George: yeah]. Now, when we move from proposed standard to full standard, the bar gets lifted quite high, multiple independent, interoperable implementations of every part of that specification. So a full standard is a promise. It says, if you quote this full standard, the implementation shall do the following, or, you know, go hit us around the head, sue us, whatever. [George: Yeah], because if it conforms to this full standard, we know. Know there is running code we know because otherwise it wouldn't have been one, and we know they interoperate those implementations. George Michaelson 5:06 Yup, and when you look at protocols that have been developed to that really strong test, it tends to have qualities that in kind of the documentation state, you minimize the use of the word may, you minimize choice behavior that leads to intractable problems. If you imagine a protocol as a complex clockwork machine, there are positions you could put the wheels in where you just can't turn the wheels anymore. That's not beneficial. So you kind of operate the machine saying, we're not going to drive to this corner. We're going to do something to get out. For instance, in BGP, across the history of BGP, we've seen people deliberately say we are now going to document how to get out of being wedged in this behavior between two parties. Geoff Huston 5:51 So despite all that, despite all that, there are still choices in full standards. [George; Oh yeah], there are still ways of setting things, and some are actually mind bogglingly stupid. Don't do that. How do we codify sensible advice about these standards? [George: Right] Well, it came up as a document series, which is kind of another RFC line called a best current practice. George Michaelson 6:16 So we have RFCs. They're just documents that the IETF publishes. We have informational, which is, this is what we think. Whatever we have experimental, which is, we did a thing Geoff Huston 6:28 There be dragons. It might not work. George Michaelson 6:30 We have proposed standard, which is, we're feeling a bit like in the future. We're heading to this point, and we're confident this is on the roadmap. And we have full standard, STD, full standard, Geoff Huston 6:43 And then we have BCPs, best current practice. George Michaelson 6:48 Ah, so this relates back to what we were discussing last time. What is the real meaning of that word, current in BCP, because we kind of touched on this. Geoff Huston 6:58 We kind of touched it, and it's kind of going should a BCP describe what we should be doing or what is available to do? And it's this kind of thing of, is a BCP a shopping list, or is it a wish list? You know, are we writing to Santa Claus, or are we heading down to the local supermarket? If I want to put it in more mundane terms, [George: yeah], because the latter, the supermarket, it's real. It's tangible. It's on the shelves. It exists. And if a BCP says, well, tick this box and not that box, then I can do that. Whereas, if I'm writing to the Easter Bunny, the answer really is, hey, rocket ship to Pluto, off we go. You know, George Michaelson 7:37 something that's been a constant across the years of doing this recording is that you have a really strong interest in economics. You've got a lovely arm chair behind you, and I think we could call you an arm chair economist in the best possible sense. It interests you. It fascinates you. And so people have beliefs about wider standards exist, and there's all this stuff about the technology and the way things work, but you've often stood up to say those things are true, but the real reason the standards exist is actually for the economic outcome of providing a basis for consumers and purchasors of equipment to go to market and buy real goods and services that behave in the right way to deliver the outcome they want. Geoff Huston 8:20 Right. It's for a way for consumers to say to an unknown set of providers. At this point, I want this function done. Well, what's the function? Well, it's RFC, 2310. It's RFC this. It's a specification of behaviors. I want you to quote me equipment, quote me solutions that do this task. And I say to the folk who wish to tender it must conform to RFCs, 1 2 3 4 5, and huge numbers of technical specifications for purchase out there in the industry quote these RFCs because, [George: oh yeah] that's the way consumers speak to providers George Michaelson 8:59 and lawyers and accountants go to these documents to say, in front of processes of mediation, this box did not meet contract, and I am not going to pay it goes directly to fights. Geoff Huston 9:11 Oh, yes, absolutely, absolutely, because that's the role of standards. They're actually there to protect the consumer from, if you will, sharp practice by a provider. Yes, this does RFC 3, when it obviously doesn't, is actually grounds of misrepresentation of what you're trying to write to. You know, supposedly sell to that consumer. RFC is grease the wheel. George Michaelson 9:32 So we're kind of heading to a direction which is arguing a case. And I feel very strongly the case being argued is the only useful meaning of BCP is a reality statement, not a wish list, because how can you deliver that certainty against the complexities of a complex technical profile that has mays and coulds and shoulds if you don't have a document that says, DO, this is what we DO. Geoff Huston 9:58 So here am I as a co. Chair of the DNS directorate, and me and my erstwhile co chair, Jim Reid, have a group of, I think about 13 or so DNS experts, and our job is to review documents that describe the DNS in various ways or relate to the use of the DNS. So it's not just DNS documents, the documents that refer to the DNS and we hand these documents to reviewers to go, does this make sense from a DNS perspective? Is what they're saying? Reasonable or not? [George: Yeah], give us a heads up. Tell us whether this document is ready or not ready. George Michaelson 10:34 These experts include people who actually have lived experience of writing code of operating services. And I don't want to imply that there's some magic I have the last say here. But if someone in a position of having written DNS systems stood up and said, I have many years experience and looking at this as a problem statement, I'm telling you, I don't see this in our code line, and I'm actually struggling to understand how I could implement that into my product. That's a massive red flag to me. That's huge. Geoff Huston 11:05 So one of these documents came up, and it was RFC 3901bis, and it's actually subtly changing the behavior of the DNS and advocates as a best current practice that if a recursive resolver that only has V6 capability is confronted with a scenario where the name servers that he was trying to reach can only do v4 then the BCP says this recursive resolver in order to get an answer, is able to forward that query to another recursive resolver, presumably that has V4. I don't know. You can't tell in advance, can you? Because recursive resolvers don't talk to each other. So this BCP, this shopping list, is advocating a response to a problem forwarding, [George: yeah], that does not exist in today's code. I can't go down to the supermarket and buy one. George Michaelson 12:06 So let's just be clear, the concept of forwarding in the DNS absolutely does exist in parts of the current DNS code available. It's a configuration element you can write into, I know bind. So I'm going to say the bind config. There's a block of text that says, when you don't know the answer, here is a list of addresses that you can send questions to so that you don't wind up having to say, I don't know. But that existence in code is tied to code paths where I'm exposed to an outcome that drives me to I'm about to say to the user, I've reached the end of my chain, and I'm going to have to say I don't know. Geoff Huston 12:46 Forwarding is when you do not have local authority, and it's an unconditional it's not part of the code, it's part of the config, George Michaelson 12:53 right config. Geoff Huston 12:54 Forwarding is also weird, because there's no TTL, there's no time to live, there's no hop count, there's no governance of forwarding, [George: right] So in the simplest question, what if I forwarded all queries to you and you forwarded all queries to me, George Michaelson 13:08 we're blowing up the network between us. Geoff Huston 13:10 We are blowing it up. And the issue is, it doesn't be a too wide loop. It could be 1020, 30, it could be an enormous loop, but it's a loop because the DNS can't detect query loops just can't so this is dangerous. Use forwarding with incredible caution. And if I was to write a BCP, I would say forwarding is dangerous. Use it when you know what you're doing. If you don't know what you're doing, this is a document to read to say, don't, just don't. [George: Yeah], whatever you're trying to do, forwarding is not the answer George Michaelson 13:39 Coming back to the code. If you're in the middle of the DNS events we were talking about and someone's proposing that they're going to construct a new belief "forward", it doesn't relate to the word forward that's in the config file that we just discussed. You're in different code logic in your DNS engine. So the idea that, Oh, we have forwarding. It's like, yeah, you have forwarding, but not in the code path that you're currently in. Geoff Huston 14:05 So the real question is, if you find a BCP or a proposed BCP advocating an action that has no code implementation, that has some real caveats as to the danger of using that particular solution that doesn't exist yet. And the reason why it doesn't exist is, you know, it is dangerous to use that kind of mode, George Michaelson 14:26 or it just hasn't been coded. I mean, it doesn't have to have all the force of danger, Geoff Huston 14:31 unconditional forwarding, or even conditional forwarding, is dangerous. George Michaelson 14:35 I agree Geoff, but the quality in BCP land, for me is dangerous or not. If it isn't there, I don't see how it could be anything but proposed standard. And I actually meant to ask you, is there any mechanistic approach that has ever taken a document in current state standard and functionally pulled it back to proposed standard defining the same behavior? Geoff Huston 14:57 Oh, in theory, I've never heard anyone. Being pulled back. What I have heard is that if a particular solution is questionable, vague, uncertain, we call it experimental. Play with this a bit. See what we think. George Michaelson 15:11 Yeah, get experience. Geoff Huston 15:13 And if you don't know what you're doing and you're inventing a new response, like Explicit Congestion Notification, step one, experimental. Check it out if it looks good. Move on. So here am I in the DNS directorate reviewing, because the chairs are also reviewers, reviewing this document, going, Is it ready for publication as a BCP? Now I go, it's a BCP. It's a shopping list. I could go down to my DNS shop and tick this box saying, I want this, yes, no, I can't. It doesn't exist. And the security considerations even say in that document, this is pretty unsafe. You need to know what you're doing in your config. This is not BCP material. George Michaelson 15:56 yeah. How can it be best if it comes with a qualifier that it has huge risks of creating traffic amplification, instability and loops. It feels unwise from a naive perspective, Geoff Huston 16:09 not naive from the perspective of the ietfs reputation. And the ietfs reputation is actually built on that ultimate pragmatism of we're not building clouds in the sky. We're not creating mountains of paperwear, about vaporware. We're creating descriptions of running code. We're creating stuff that consumers can use be their massive ISPs, or you and me at home in our supply chains, with our focus selling equipment to if you will assure ourselves of the functional capability of the equipment and technology we're using. And as soon as aspirational wish lists enter that sphere, the IETF has dropped off the cliff and plugged into an abyss of simply undermining its own credibility, evangelism gets into the path of practical value, realism, [George yeah], as soon as you give in to the V6 evangelists in this all hope is lost of the credibility the organization pushing it, don't go there. George Michaelson 17:13 I have different feelings depending on the day of the week about the problem space. Geoff. I want to say up front that I do think of myself as a V6 evangelist. I do, but I have a very strong streak of pragmatism. And the quality about the discussion we're having now is that when it's time to evangelize, there's a prescriptive label for your document that we all know and understand. It's experimental and it's proposed standard at the fringes, when you have higher confidence, that's the space, that you could actually go to vendors and say, this technology is the future, and if you invest now, there's potential for an upside, and they can legitimately come back and say, my banking community is not remotely interested in that, and I'm going to Keep doing this other thing, because IBM bit net protocol is what I want to do. Well, fine, that's a rational choice for them, but it's where you go in the evangelism state. When you move to an operational dependency, my mind kind of flips. And as a consumer, I look at things like, wow, the ambulance service is using this now. Wow. The geolocation service the ambulance service uses to do this depends on the DNS to do its bootstrapping. If I put a service out that has a 40 or a 50 or a 20% failure, and that then means the emergency services actually can't perform where is this customer to direct an ambulance? I've potentially specked up standards that killed people. That's the level of pragmatism I have about behavior. Here we've become infrastructure. We're talking about water quality, gas pressure, electricity, voltage stability. Geoff Huston 18:54 So what would you say is that BCPs and full standards, full standards and BCPs are important George Michaelson 19:02 Yes, Geoff Huston 19:02 And people are relying on the promise George Michaelson 19:06 Yes Geoff Huston 19:06 Of running code, that this stuff works and is known to work as described on the team. You know this works. George Michaelson 19:15 That is absolutely how I feel about this behavior. Geoff, that's it. Geoff Huston 19:20 I put a label on my review, going, not ready, right? It's not a BCP. George Michaelson 19:24 This is a process component that you are in the DNS Directorate. You also have some engagement in the working group. You have a functional role to declare. I don't feel comfortable with this thing in a formalism that is about late stage review, because it's a given at this point, the working group, for whatever reason, has decided they want to put this forward. Geoff Huston 19:46 I'm not sure which working group, because it straddles DNS and V6, but even so, there are aspects of this DNS behavior described in this BCP that do not exist and are unsafe in terms of a practice, [George: right] So. Certainly do not, in my mind, form the pragmatic shopping list of reliable configurations of standard technology that a BCP should normally do, [George: right] But I'm just a reviewer who holds the key, who holds the IETF reputation. [George: Yes], well, interestingly, it's the steering group, the Internet Engineering Steering, and these folk, two from each of the areas, really, most cases, are really the holders of that IETF quality promise to the world in general, if they're the final gating function that says this document shall be published or not. And yet, think when someone says this is not ready, because it's not a BCP, because the view of a BCP is x. I'm finding it amazing to go and hear feedback from some of the AD's, the area directors going now you misunderstand a BCP. A BCP is best practice, best in an aspirational sense, current is what our best hopes today are. So I'm perfectly comfortable in a document that is unpurchasable, a specification that does not exist, that might even be unsafe, but it accurately reflects today's aspirations. And I sit there going, Wow, that's an inventive use of English. I've never heard it like that before. [George: Yeah] we should not be going there. George Michaelson 21:19 I feel that's an inversion of prior behavior that we had established over many, many years. It's as if a side effect of perhaps the renewal process, because these bodies aren't it's not a job for life. Geoff, right? You move through Right. And so I'm beginning to wonder if, in the move through we've had a cultural shift that hasn't been adequately discussed and communicated. I'd like to say at this point that you have had a functional role in this. In times past, you have been on the IESG, you're not now. You've been on the IAB, you have been the notional head of the pyramid of formal behavior. So you've had the functional role. I currently have a very silly, small functional role on the board that runs the infrastructure, and there's kind of a giant Chinese wall that I don't have a function in this side of the business. This isn't the part of the business that concerns me, because my job is to make the machinery run. But I'm also a pragmatic human being, and I depend on the global Internet, and when I hear that the standards body is starting to discuss approaches to defining things as BCP or STD that wouldn't align with rational purchasing, with safe operational state Geoff Huston 22:35 Align with running with available code, George Michaelson 22:37 that's not good. Geoff Huston 22:38 That's really debating the value of the IETF and its outcomes, and it's demeaning it in a way that you kind of look at all of them going, well, I know about the DNS, but what about MPLS? What about this? What about that? George Michaelson 22:51 Yeah, is this happening in other places? Geoff Huston 22:54 Exactly once you find one error in a large compendium of information, the whole collection becomes a little bit devalued because you don't know if there are others, and you don't know where they might be. [George: Yeah] and I find that behavior for a rigorous group that the IETF must necessarily be [George: worrying]. I find that laxity of behavior and interpretation around these documents not only just worrying, but I think unacceptable. George Michaelson 23:21 Yeah, I've noticed there's been a discussion happening on the IETF general list where people have been saying, I'm observing that people are forum shopping, and they're taking their documents to other forum because they don't see progress happening inside this group. And some responses are very directed to outcome, oh, we must improve our processes. We must make it easier for people to get documents through working group process. But I am more aligned with voices that are saying, if you're not comfortable that your idea gets traction in our forum, the answer for us is not necessarily that we are doing anything wrong. We might want to be saying, yes, there's a reason those ideas don't come to the surface. Geoff Huston 24:05 We apply a known set of standards behaviors to our product, and our reputation is based on the consistent, and I would almost say, rigid application of those standards to the product. And once we falter in that once we acquiesce to group think, Oh, the V6 folk are trying hard. Oh, you know, the authors are great folk. You know, they deserve to have an RFC done. Once you actually fall for that, then the cost is in the reputation of the IETF as a whole. Because the IETF reputation is based on the rigidity and robustness of its processes in ensuring the quality of its outcomes. No one is an expert in everything the IETF does. No one [George: No] but as consumers, we expect that if we can quote an RFC, it will do the job it's intended to do, and once you get the inkling of a suspicion that's not going on. Then I think we're all the poorer for it, and that's a bad place to be. So in this case, I suppose I'm calling out the IESG going, guys, girls do better, [George: yeah] be more rigid. And if you find, oh, the job is too hard, there are others who are willing to do this. George Michaelson 25:15 Yeah, learning to say no is a very, very important skill in these activities. Geoff Huston 25:21 Don't just bend with everyone else and bend to oh, well, it's okay. Everyone else has said, good. It's good. You know, there are times when it's really this is not good, and not because of the quality of the document. It's what it's trying to propose and the attribution that it is going to have. [George: Yeah] this is a shopping list of a reliable, available technology. It is not a wish list sent to the Easter Bunny. George Michaelson 25:44 This is a hard place to be Geoff, because I feel like you, this is something that needs to be said, but it's very hard to see how people are going to get out of the situation they've put themselves in. Geoff Huston 25:56 I'm not on the IESG George, and I'm glad I don't have to resolve this, but I would certainly urge them to think long and hard about that document series and new entrance into it and what its role is. Because if some folk on the IESG have a different interpretation that needs to be hammered out, [George: yeah], and perhaps even its own BCP or full standard to say this is what a BCP is yes and what it is not. George Michaelson 26:23 I think that's going to be a very important conversation. Geoff, thank you for bringing this one up. That's been really interesting. Thank you. Geoff Huston 26:31 Oh, thank you, George, thanks. George Michaelson 26:33 If you've got a story or research to share here on PING, why not get in contact by email to ping@apnic.net or via the APNIC social media channels, also remember the measurement@apnic.net mailing list on orbit is there to discuss and share relevant collaborative opportunities, grants and funding opportunities, jobs and graduate placings, or to seek feedback from The community on your own measurement projects, be sure to check out the APNIC website for all your resource and community needs until next time you.