1 00:00:01,760 --> 00:00:03,699 Join us as we gather around the hedge, 2 00:00:03,759 --> 00:00:05,139 where we dig into technology, 3 00:00:05,440 --> 00:00:08,160 business, and culture with the finest minds in 4 00:00:08,160 --> 00:00:09,139 computer networking. 5 00:00:20,414 --> 00:00:21,554 Good morning, Tom. 6 00:00:21,934 --> 00:00:22,994 Good morning, Russ. 7 00:00:23,614 --> 00:00:25,234 There are still holes in your bookshelf. 8 00:00:26,109 --> 00:00:26,609 Yep. 9 00:00:26,910 --> 00:00:27,869 Yep. And There 10 00:00:28,589 --> 00:00:30,509 it's a sign, Russ. You need to write, 11 00:00:30,509 --> 00:00:32,189 like, you need to write, like, five more 12 00:00:32,189 --> 00:00:32,689 books. 13 00:00:35,390 --> 00:00:37,309 In my spare time. That's what those holes 14 00:00:37,309 --> 00:00:39,969 are for. Yeah. Yeah. In my spare time. 15 00:00:40,085 --> 00:00:42,405 Yeah. I don't know. I I probably do 16 00:00:42,405 --> 00:00:43,844 need to write more books, but I just 17 00:00:43,844 --> 00:00:45,844 need to write more. But this has been 18 00:00:45,844 --> 00:00:47,844 a busy this has been a really busy 19 00:00:47,844 --> 00:00:49,225 six months. I don't know. 20 00:00:49,604 --> 00:00:51,125 I I for those who don't know, I 21 00:00:51,125 --> 00:00:53,045 teach at University of Colorado once a year, 22 00:00:53,045 --> 00:00:54,565 and I'm in the middle of University of 23 00:00:54,565 --> 00:00:56,670 Colorado. And I'm also in the middle of 24 00:00:56,670 --> 00:00:57,329 that course. 25 00:00:57,789 --> 00:01:00,270 And I'm also doing a bunch of recording 26 00:01:00,270 --> 00:01:02,590 for a training thing that I'm doing, and 27 00:01:02,590 --> 00:01:05,090 I'm trying to develop new material for Pearson 28 00:01:05,150 --> 00:01:07,150 and, you know, just and, and, and, and, 29 00:01:07,150 --> 00:01:08,909 and. So springs are always 30 00:01:09,594 --> 00:01:11,935 spring is always, for some reason, hyper busy. 31 00:01:11,994 --> 00:01:13,994 It it's, you know, it's like I wanna 32 00:01:13,994 --> 00:01:15,754 shift some stuff to fall, but so far 33 00:01:15,754 --> 00:01:18,015 I've figured out how to. So okay. 34 00:01:18,474 --> 00:01:20,155 Well, you can okay. You have my permission 35 00:01:20,155 --> 00:01:21,834 to wait to write those five books until 36 00:01:21,834 --> 00:01:22,494 the fall. 37 00:01:23,159 --> 00:01:25,640 Okay. The holes in your bookshelf will wait 38 00:01:25,640 --> 00:01:27,319 till fall. I see how that's gonna work. 39 00:01:27,319 --> 00:01:28,140 Okay. Great. 40 00:01:28,920 --> 00:01:31,099 And today, we are joined by Dell Darryl 41 00:01:31,479 --> 00:01:34,119 Swear? Swear? How do you I'm sorry. Swear. 42 00:01:34,119 --> 00:01:37,064 That's right. Yeah. Swear. Darryl Swear. Yeah. Alright. 43 00:01:37,064 --> 00:01:39,465 And where are you, Daryl, physically? Where do 44 00:01:39,465 --> 00:01:40,765 you where do you hail from? 45 00:01:41,465 --> 00:01:44,265 Yeah. So I hail from Shillong. It's a 46 00:01:44,265 --> 00:01:47,484 small city, small town up in Northeastern India, 47 00:01:47,864 --> 00:01:51,084 and that's where I currently am physically located. 48 00:01:51,625 --> 00:01:52,125 So, 49 00:01:52,599 --> 00:01:53,819 yeah. That's where I'm from. 50 00:01:54,200 --> 00:01:56,759 Okay. Cool. So close to home. I'm actually 51 00:01:56,759 --> 00:01:58,039 not that far from home. I grew up 52 00:01:58,039 --> 00:01:59,959 in the Southeast, and I've never moved very 53 00:01:59,959 --> 00:02:00,939 far out of it. 54 00:02:01,479 --> 00:02:03,799 The furthest I've actually lived from from my 55 00:02:03,799 --> 00:02:05,984 home, from my original from, like, where I 56 00:02:05,984 --> 00:02:08,004 grew up is actually New Jersey. 57 00:02:08,865 --> 00:02:09,685 Isn't that crazy? 58 00:02:10,064 --> 00:02:11,504 Like, I've stayed on the East Coast my 59 00:02:11,504 --> 00:02:12,884 entire life. Somehow, 60 00:02:13,344 --> 00:02:14,645 I've been in technology 61 00:02:15,185 --> 00:02:16,944 for thirty years, and I've never moved to 62 00:02:16,944 --> 00:02:17,444 California. 63 00:02:18,360 --> 00:02:19,960 I don't know if that's a you should 64 00:02:19,960 --> 00:02:21,740 feel sorry for me or if that's 65 00:02:22,760 --> 00:02:23,260 a 66 00:02:24,840 --> 00:02:27,000 I'm not sure what I'm not sure which 67 00:02:27,000 --> 00:02:27,819 that should be. 68 00:02:28,280 --> 00:02:30,439 So let's talk about out of band networks. 69 00:02:30,439 --> 00:02:30,939 Right? 70 00:02:31,319 --> 00:02:32,540 AWV. So 71 00:02:33,485 --> 00:02:35,805 out of band networks are a thing. And 72 00:02:35,805 --> 00:02:38,044 I know that, for instance, when I was 73 00:02:38,044 --> 00:02:39,564 at LinkedIn, we had an out of band 74 00:02:39,564 --> 00:02:40,465 management network. 75 00:02:40,925 --> 00:02:43,004 And I also know that we got rid 76 00:02:43,004 --> 00:02:43,585 of it 77 00:02:43,965 --> 00:02:44,939 to some degree. 78 00:02:45,260 --> 00:02:46,240 We we had 79 00:02:46,699 --> 00:02:48,219 it, but we had it we we moved 80 00:02:48,219 --> 00:02:50,460 it from being its own separate infrastructure, physical 81 00:02:50,460 --> 00:02:50,960 infrastructure, 82 00:02:51,419 --> 00:02:52,159 to being, 83 00:02:52,860 --> 00:02:53,520 a verf. 84 00:02:54,379 --> 00:02:56,240 So it still had its own IP addressing, 85 00:02:56,300 --> 00:02:58,159 but it was on the same physical infrastructure. 86 00:02:59,125 --> 00:03:01,284 But I don't know of anybody else that 87 00:03:01,284 --> 00:03:02,905 I've worked on their network. 88 00:03:03,205 --> 00:03:05,604 I can't think of another single network I've 89 00:03:05,604 --> 00:03:07,544 ever worked on that actually 90 00:03:08,164 --> 00:03:10,164 that that I have managed or been in 91 00:03:10,164 --> 00:03:12,004 the part of management. I've seen them in 92 00:03:12,004 --> 00:03:12,824 other networks 93 00:03:13,830 --> 00:03:15,769 where there's actually an out of band network. 94 00:03:16,150 --> 00:03:17,209 Isn't that crazy? 95 00:03:17,909 --> 00:03:19,590 Like, you would think it's the most natural 96 00:03:19,590 --> 00:03:20,949 thing in the world to have an out 97 00:03:20,949 --> 00:03:22,729 of band network. Why why not? 98 00:03:23,349 --> 00:03:23,849 So 99 00:03:24,229 --> 00:03:26,789 yeah. You know? So so, Darryl, what's your 100 00:03:26,789 --> 00:03:28,310 take on this? Like, where where do you 101 00:03:28,310 --> 00:03:31,004 start thinking about it? In in my opinion 102 00:03:31,004 --> 00:03:33,085 and in my in my experience, right, you 103 00:03:33,245 --> 00:03:35,905 I agree with your observation that most, 104 00:03:36,284 --> 00:03:38,844 networks out there, regardless if there's service provider 105 00:03:38,844 --> 00:03:41,405 networks or data center networks or enterprise or 106 00:03:41,405 --> 00:03:42,385 something in between, 107 00:03:43,485 --> 00:03:44,465 most of them, 108 00:03:44,830 --> 00:03:46,990 for reasons I cannot fathom, they do not 109 00:03:46,990 --> 00:03:48,989 have an out of band network, let alone 110 00:03:48,989 --> 00:03:50,689 let alone an in band management 111 00:03:51,229 --> 00:03:53,150 framework of some sort. A lot of these, 112 00:03:53,389 --> 00:03:55,250 networks out there, they they don't have 113 00:03:55,629 --> 00:03:56,129 a 114 00:03:56,590 --> 00:03:57,090 streamline, 115 00:03:57,789 --> 00:03:59,009 management implementation, 116 00:03:59,310 --> 00:04:01,715 right, let alone an auto band network. 117 00:04:02,014 --> 00:04:05,215 And I think that's a bad position to 118 00:04:05,215 --> 00:04:06,834 be in because when 119 00:04:07,294 --> 00:04:09,555 disasters happen, when things break, 120 00:04:10,254 --> 00:04:12,509 whether that's, you know, whether that's a layer 121 00:04:12,509 --> 00:04:15,169 two plus or whether that's a physical disaster, 122 00:04:15,229 --> 00:04:18,370 layer one and below, including potentially layer zero. 123 00:04:19,789 --> 00:04:22,110 You definitely should have an outbound in place 124 00:04:22,110 --> 00:04:23,410 in some shape or form, 125 00:04:23,949 --> 00:04:26,509 to ensure that you can restore connectivity and 126 00:04:26,509 --> 00:04:30,354 access to your relevant systems and devices and 127 00:04:30,354 --> 00:04:32,435 so on and so forth. Right? Like, in 128 00:04:32,435 --> 00:04:35,414 the past, five, six years, we've seen quite, 129 00:04:36,274 --> 00:04:37,475 a few number of, 130 00:04:37,875 --> 00:04:40,675 large scale outages. Right? Like, there was the 131 00:04:40,675 --> 00:04:41,175 famous, 132 00:04:41,949 --> 00:04:45,470 Facebook slash meta outage back in 2020, or 133 00:04:45,470 --> 00:04:47,250 was it '21? I don't remember. 134 00:04:47,709 --> 00:04:50,850 There was the Optus out outage in Australia, 135 00:04:51,550 --> 00:04:53,870 and there there was various other outages that 136 00:04:53,870 --> 00:04:55,410 I I can't recall right now. 137 00:04:55,774 --> 00:04:58,414 And from what we can tell, based on 138 00:04:58,414 --> 00:05:00,914 the public information on some of these 139 00:05:01,294 --> 00:05:02,675 instance, these outages, 140 00:05:03,294 --> 00:05:06,175 it wouldn't be very difficult for somebody to 141 00:05:06,175 --> 00:05:07,794 gauge that if these, 142 00:05:08,419 --> 00:05:08,919 instances 143 00:05:09,459 --> 00:05:12,519 had a properly architected out of band network, 144 00:05:13,379 --> 00:05:15,479 those disasters could have been mitigated 145 00:05:15,779 --> 00:05:18,180 pretty much a lot faster than they were 146 00:05:18,180 --> 00:05:20,974 because, they didn't have a sufficient 147 00:05:21,594 --> 00:05:24,175 or properly architected out of band network. So, 148 00:05:25,354 --> 00:05:26,954 yeah, it's a mystery to me as well, 149 00:05:26,954 --> 00:05:28,954 Tom. I have no idea why most people, 150 00:05:29,354 --> 00:05:31,294 don't take out of band like a, 151 00:05:31,995 --> 00:05:34,154 natural first step to building a network. For 152 00:05:34,154 --> 00:05:34,894 me personally, 153 00:05:36,040 --> 00:05:39,079 for the greenfield networks that I've worked on, 154 00:05:39,079 --> 00:05:40,680 my first step is building the out of 155 00:05:40,680 --> 00:05:43,000 band network. I I don't build the edge 156 00:05:43,000 --> 00:05:45,160 or the core or the aggregation and access. 157 00:05:45,160 --> 00:05:47,560 No. First step's always gonna be the out 158 00:05:47,560 --> 00:05:49,879 of band network because then that's how I 159 00:05:49,879 --> 00:05:52,975 myself, along with the staff or people who 160 00:05:52,975 --> 00:05:55,134 are in charge of the network will actually 161 00:05:55,134 --> 00:05:57,454 log in and make changes to the rest 162 00:05:57,454 --> 00:05:59,314 of the network through the auto band. 163 00:06:00,014 --> 00:06:02,274 So that's how I personally see it, but, 164 00:06:03,134 --> 00:06:04,735 I do I do wish and hope more 165 00:06:04,735 --> 00:06:06,735 people would adopt an auto band approach at 166 00:06:06,735 --> 00:06:08,910 least some extent, for sure. There's actually a 167 00:06:08,910 --> 00:06:11,470 funny story from Cisco Tech about this. We 168 00:06:11,470 --> 00:06:12,289 used to have, 169 00:06:12,910 --> 00:06:13,649 ship kits, 170 00:06:15,389 --> 00:06:17,709 or mail kits. And what it was was 171 00:06:17,709 --> 00:06:19,569 it was basically a hard case 172 00:06:19,870 --> 00:06:20,769 with a modem 173 00:06:21,314 --> 00:06:22,855 and all the correct cables. 174 00:06:23,395 --> 00:06:25,634 And if you lost access to your router 175 00:06:25,634 --> 00:06:26,855 and you called TAC, 176 00:06:27,395 --> 00:06:28,995 we would ask you for an address, and 177 00:06:28,995 --> 00:06:30,375 we would overnight FedEx 178 00:06:31,235 --> 00:06:33,555 a ship kit to it and ask if 179 00:06:33,555 --> 00:06:35,495 there was anybody physically on-site 180 00:06:35,970 --> 00:06:38,930 who could just, you know, find the right 181 00:06:38,930 --> 00:06:40,230 cables that would fit 182 00:06:40,610 --> 00:06:42,370 and put it all in there. And it 183 00:06:42,370 --> 00:06:44,689 was like a modem and everything else and 184 00:06:44,689 --> 00:06:47,009 just please connect the telephone line to it. 185 00:06:47,009 --> 00:06:49,490 And later, they became five g or four 186 00:06:49,490 --> 00:06:51,589 g, not five g, but four g. Right? 187 00:06:51,774 --> 00:06:53,454 Later, they became it was a four g 188 00:06:53,454 --> 00:06:54,194 cell phone 189 00:06:54,654 --> 00:06:56,495 with the right cable connectors and a much 190 00:06:56,495 --> 00:06:58,894 smaller box because there was no modem. And, 191 00:06:58,894 --> 00:07:01,074 like, you would just connect everything together, 192 00:07:01,375 --> 00:07:03,294 and then we could, from the tech, get 193 00:07:03,294 --> 00:07:03,794 in 194 00:07:04,149 --> 00:07:06,149 and go get you access back to your 195 00:07:06,149 --> 00:07:08,410 router. But this see, that that seems ridiculous 196 00:07:08,470 --> 00:07:10,470 to me, because you're at the point in 197 00:07:10,470 --> 00:07:13,110 your network where you have to ship a 198 00:07:13,110 --> 00:07:14,250 physical box 199 00:07:16,709 --> 00:07:19,050 to get back in. That's that's just crazy. 200 00:07:19,615 --> 00:07:21,074 I I think a lot of times, 201 00:07:21,694 --> 00:07:24,115 like human nature works against us here. 202 00:07:24,495 --> 00:07:26,574 You build you build networks, and especially if 203 00:07:26,574 --> 00:07:28,735 it's a if it's a a big network 204 00:07:28,735 --> 00:07:29,954 that generates revenue, 205 00:07:30,574 --> 00:07:32,914 the the network itself has to be resilient. 206 00:07:33,055 --> 00:07:35,149 And so I think a lot of times, 207 00:07:35,149 --> 00:07:38,110 it's just really it's it's easy to overlook 208 00:07:38,110 --> 00:07:39,870 out of band because, well, I've I've built 209 00:07:39,870 --> 00:07:41,389 the network to be resilient. If I have 210 00:07:41,389 --> 00:07:43,550 a failure here, I've built it to route 211 00:07:43,550 --> 00:07:46,269 around this problem. And so we we lean 212 00:07:46,269 --> 00:07:48,449 on the resiliency of the network to, 213 00:07:49,245 --> 00:07:52,604 kind of, you know, where management management traffic 214 00:07:52,604 --> 00:07:54,365 goes up the wrong direction or around the 215 00:07:54,365 --> 00:07:55,264 horn or whatever, 216 00:07:55,725 --> 00:07:57,564 we just we just it's like, well, we 217 00:07:57,564 --> 00:07:59,504 invested in this resiliency, so, 218 00:07:59,964 --> 00:08:01,884 it's not a pressing issue right now. And 219 00:08:01,884 --> 00:08:03,870 I think for for most I would say 220 00:08:03,870 --> 00:08:05,790 for most operations, there's, like, a whole bunch 221 00:08:05,790 --> 00:08:07,949 of things that are fires that are burning 222 00:08:07,949 --> 00:08:09,889 hotter than that. And then, unfortunately, 223 00:08:10,990 --> 00:08:12,210 what happens is, 224 00:08:12,830 --> 00:08:14,830 eventually an outage comes along that you didn't 225 00:08:14,830 --> 00:08:16,590 anticipate in the in the design of your 226 00:08:16,590 --> 00:08:19,485 network, And, the resiliency that you built didn't 227 00:08:19,485 --> 00:08:22,285 cover didn't cover this case, and so you 228 00:08:22,285 --> 00:08:23,884 see some of the outages like the ones 229 00:08:23,884 --> 00:08:25,964 you were talking about, Daryl. But, I think 230 00:08:25,964 --> 00:08:27,644 it's just it's just hard to invest in 231 00:08:27,644 --> 00:08:29,504 something. It's it's like paying insurance. 232 00:08:30,520 --> 00:08:31,560 You know, I think there are a lot 233 00:08:31,560 --> 00:08:32,919 of people, at least in The United States, 234 00:08:32,919 --> 00:08:34,440 that wouldn't pay any insurance at all if 235 00:08:34,440 --> 00:08:35,980 they weren't forced to by law. 236 00:08:36,440 --> 00:08:39,000 Because we don't wanna we don't wanna pay 237 00:08:39,000 --> 00:08:41,240 for things that we don't that aren't hurting 238 00:08:41,240 --> 00:08:42,120 us right now. 239 00:08:42,600 --> 00:08:44,299 So it's kind of a human nature thing. 240 00:08:44,884 --> 00:08:47,625 And it's not just cost. It's management. Right? 241 00:08:47,764 --> 00:08:50,565 This entire out of band network needs to 242 00:08:50,565 --> 00:08:51,465 be automated, 243 00:08:52,085 --> 00:08:54,424 configured. You need IP address space. 244 00:08:54,725 --> 00:08:55,865 You need tools. 245 00:08:56,165 --> 00:08:58,004 You need to monitor it and make sure 246 00:08:58,004 --> 00:08:58,825 it's working. 247 00:08:59,500 --> 00:09:00,779 Because a lot of times, it's easy to 248 00:09:00,779 --> 00:09:02,480 put an out of band network in place 249 00:09:02,620 --> 00:09:04,220 and then never use it. And then when 250 00:09:04,220 --> 00:09:05,579 you get to the point where you need 251 00:09:05,579 --> 00:09:08,220 it, oh, that doesn't work. Oh, oh, that's 252 00:09:08,220 --> 00:09:08,879 a problem. 253 00:09:09,339 --> 00:09:12,379 Right? Because you never use it. So I 254 00:09:12,379 --> 00:09:13,919 don't know. I mean, there's 255 00:09:14,945 --> 00:09:17,605 there's lots of stuff around this area. 256 00:09:19,504 --> 00:09:21,684 So is there any other justification? 257 00:09:22,384 --> 00:09:24,304 I I have justifications, but I'll I'll ask 258 00:09:24,304 --> 00:09:25,524 if you all think of any. 259 00:09:26,384 --> 00:09:29,429 Is there any other justification besides the, oh, 260 00:09:29,429 --> 00:09:30,950 my network is down. I can't get to 261 00:09:30,950 --> 00:09:33,610 this site or this host or this router 262 00:09:34,230 --> 00:09:35,690 for an out of band network? 263 00:09:36,149 --> 00:09:37,909 Like, what else if I was going to 264 00:09:37,909 --> 00:09:40,710 my boss and saying, we should spend money 265 00:09:40,710 --> 00:09:42,169 on an out of band network. 266 00:09:43,235 --> 00:09:45,475 What what what should I say? How should 267 00:09:45,475 --> 00:09:46,375 I like, 268 00:09:46,835 --> 00:09:48,294 beyond, oh, there's a catastrophe 269 00:09:48,595 --> 00:09:50,595 waiting in the wings. I have an idea 270 00:09:50,595 --> 00:09:52,674 too, but I wanna hear from Daryl. Yeah. 271 00:09:52,674 --> 00:09:53,174 Yeah. 272 00:09:53,634 --> 00:09:54,134 So 273 00:09:55,570 --> 00:09:57,410 I use it now. So there was this, 274 00:09:57,730 --> 00:09:59,350 there was a client of mine, 275 00:10:01,250 --> 00:10:02,470 that wanted to, 276 00:10:03,009 --> 00:10:04,690 this client of mine, he, he, he was 277 00:10:04,690 --> 00:10:06,389 a business owner, right? He was a businessman, 278 00:10:06,450 --> 00:10:08,210 but he was not a, an engineer by 279 00:10:08,210 --> 00:10:10,915 profession. Right. So he wanted me to explain 280 00:10:10,915 --> 00:10:11,575 to him, 281 00:10:13,154 --> 00:10:13,815 the importance 282 00:10:14,434 --> 00:10:16,674 of an out of band network beyond just, 283 00:10:16,674 --> 00:10:19,154 you know, disaster recovery, like like you just 284 00:10:19,154 --> 00:10:19,654 said. 285 00:10:20,035 --> 00:10:22,035 And I explained it to him this way, 286 00:10:22,035 --> 00:10:22,535 like, 287 00:10:23,210 --> 00:10:25,149 the production network would 288 00:10:25,690 --> 00:10:27,950 be similar to a public highway, 289 00:10:28,410 --> 00:10:31,710 whereas the outbound network would be similar to 290 00:10:31,769 --> 00:10:33,070 maintenance tunnels 291 00:10:33,450 --> 00:10:35,529 behind or on the side of the highway. 292 00:10:35,529 --> 00:10:37,149 Or, for example, if you have, 293 00:10:37,634 --> 00:10:40,915 public water systems, public sewage systems, there's bound 294 00:10:40,915 --> 00:10:44,274 to be maintenance tunnels or maintenance systems in 295 00:10:44,274 --> 00:10:45,254 place, right, for 296 00:10:45,634 --> 00:10:46,535 those public, 297 00:10:46,995 --> 00:10:48,215 infrastructure and services. 298 00:10:48,995 --> 00:10:51,075 And that is how I try explaining it 299 00:10:51,075 --> 00:10:52,855 to to this client of mine 300 00:10:53,210 --> 00:10:54,509 couple couple years ago. 301 00:10:54,889 --> 00:10:58,090 And he liked the, the analogy, the the 302 00:10:58,090 --> 00:11:00,970 the ID concept behind it. And it's not 303 00:11:00,970 --> 00:11:03,610 just for disaster recovery. When when I build 304 00:11:03,610 --> 00:11:05,230 an outbound network, I 305 00:11:05,929 --> 00:11:06,429 encourage 306 00:11:07,065 --> 00:11:07,565 the 307 00:11:08,024 --> 00:11:10,605 the the humans behind the network to 308 00:11:11,304 --> 00:11:11,804 exclusively 309 00:11:12,184 --> 00:11:14,125 access their remote devices 310 00:11:14,585 --> 00:11:17,165 and systems and software applications, etcetera, 311 00:11:18,184 --> 00:11:21,065 exclusively, again, truly out of band networks. So, 312 00:11:21,065 --> 00:11:21,565 essentially, 313 00:11:22,460 --> 00:11:25,440 the only point of ingress into the management 314 00:11:25,580 --> 00:11:27,200 plane of the organization 315 00:11:28,059 --> 00:11:30,299 is exclusively through the out of band network 316 00:11:30,299 --> 00:11:32,620 for 99% of the time. Now now there 317 00:11:32,620 --> 00:11:33,120 may 318 00:11:33,899 --> 00:11:35,820 be edge cases. There's always gonna be some 319 00:11:35,820 --> 00:11:37,995 edge cases or some edge corners here and 320 00:11:37,995 --> 00:11:40,315 there where this approach will not work. There's 321 00:11:40,315 --> 00:11:42,095 always gonna be that 1% exception. 322 00:11:42,475 --> 00:11:43,675 But for 99 323 00:11:43,675 --> 00:11:45,995 of the time when, when I built the 324 00:11:45,995 --> 00:11:47,055 out of band networks, 325 00:11:47,514 --> 00:11:50,014 for these very very use cases, 326 00:11:50,475 --> 00:11:53,250 I always encourage that it's supposed to be 327 00:11:53,250 --> 00:11:56,209 used by employees all the time. So if 328 00:11:56,209 --> 00:11:56,709 you 329 00:11:57,089 --> 00:11:59,169 if you go for a VPN model, the 330 00:11:59,169 --> 00:12:01,509 VPN will be behind the outbound network. 331 00:12:02,049 --> 00:12:03,250 If you go for the, 332 00:12:03,970 --> 00:12:06,315 the, what's it called again? A SOCKS proxy 333 00:12:06,315 --> 00:12:08,235 model, there there are some company that don't 334 00:12:08,235 --> 00:12:10,335 do VPNs. They prefer SOCKS 335 00:12:10,795 --> 00:12:11,934 proxying and 336 00:12:12,235 --> 00:12:14,715 zero trust and all those fancy layer seven 337 00:12:14,715 --> 00:12:16,975 based login and security measures, etcetera. 338 00:12:17,514 --> 00:12:19,800 All of that goes through the outbound network. 339 00:12:19,800 --> 00:12:22,540 It the the only ingress point for authorized 340 00:12:22,840 --> 00:12:26,940 staff, employees, and management into the network infrastructure 341 00:12:27,399 --> 00:12:30,120 or any software applications running behind it, for 342 00:12:30,120 --> 00:12:30,620 example, 343 00:12:31,095 --> 00:12:31,595 NetBox, 344 00:12:32,134 --> 00:12:34,774 LibreNMS, you know, some of these applications. The 345 00:12:34,774 --> 00:12:36,534 only way for a human to access those 346 00:12:36,534 --> 00:12:37,034 applications 347 00:12:37,654 --> 00:12:39,254 would be through the out of band network 348 00:12:39,254 --> 00:12:42,214 because it also helps with network segmentation, you 349 00:12:42,214 --> 00:12:44,315 know, keeping the production revenue network 350 00:12:44,800 --> 00:12:47,360 in a different segment, keeping the management network 351 00:12:47,360 --> 00:12:49,600 slash out of band in a separate segment. 352 00:12:49,600 --> 00:12:51,940 It also helps with the security posture, 353 00:12:52,559 --> 00:12:54,559 in in my personal opinion. I don't think 354 00:12:54,559 --> 00:12:56,259 it's a good idea to mix 355 00:12:56,879 --> 00:12:57,379 your 356 00:12:57,680 --> 00:12:58,899 management traffic 357 00:12:59,214 --> 00:13:01,375 with your customer traffic. I don't see a 358 00:13:01,375 --> 00:13:03,774 good reason to be doing that. And since 359 00:13:03,774 --> 00:13:05,534 if you have an out of band network 360 00:13:05,534 --> 00:13:06,274 in place 361 00:13:06,654 --> 00:13:08,815 and you have your network devices in the 362 00:13:08,815 --> 00:13:10,575 production network, for example, let's say you have 363 00:13:10,575 --> 00:13:12,434 a router in production network and, 364 00:13:12,910 --> 00:13:15,149 it has a management Ethernet port, not to 365 00:13:15,149 --> 00:13:17,389 be conflated with the console port, but a 366 00:13:17,389 --> 00:13:18,769 management Ethernet port. 367 00:13:19,389 --> 00:13:21,330 So what we can do from a security 368 00:13:21,389 --> 00:13:24,830 standpoint is to ensure that your management related 369 00:13:24,830 --> 00:13:25,330 applications, 370 00:13:26,084 --> 00:13:28,504 such as API access, SSH, 371 00:13:28,964 --> 00:13:31,845 streaming telemetry, etcetera, all of that should be 372 00:13:31,845 --> 00:13:32,345 configured 373 00:13:32,804 --> 00:13:35,544 to hard bind onto the management interface 374 00:13:36,164 --> 00:13:38,904 slash management v r f to ensure that 375 00:13:39,069 --> 00:13:41,870 only the management network slash OOB can access 376 00:13:41,870 --> 00:13:42,610 those services. 377 00:13:43,069 --> 00:13:45,870 That way, ensuring that nobody on the transit 378 00:13:45,870 --> 00:13:47,889 path or the public facing, 379 00:13:48,990 --> 00:13:51,089 routing table of customers, etcetera, 380 00:13:51,470 --> 00:13:53,309 will ever be able to talk through the 381 00:13:53,309 --> 00:13:54,529 daemons listening 382 00:13:54,865 --> 00:13:56,945 behind that management of VRF. So it it 383 00:13:56,945 --> 00:13:59,024 really I think it really helps with security 384 00:13:59,024 --> 00:14:02,065 and segmentation for for everybody to have some 385 00:14:02,065 --> 00:14:04,264 form of outbound network, not just to act 386 00:14:04,384 --> 00:14:06,865 to use it when there's a disaster, but 387 00:14:06,865 --> 00:14:07,605 to proactively 388 00:14:07,985 --> 00:14:11,259 use it to manage, automate, and orchestrate the 389 00:14:11,259 --> 00:14:11,759 network. 390 00:14:12,220 --> 00:14:12,960 So so 391 00:14:14,059 --> 00:14:17,040 you're actually on board with not a physical 392 00:14:17,740 --> 00:14:20,460 network, but rather a VPN or proxy based 393 00:14:20,460 --> 00:14:22,379 network as well as an out of band. 394 00:14:22,379 --> 00:14:22,815 Right? 395 00:14:23,294 --> 00:14:25,534 That's that what do you prefer do you 396 00:14:25,534 --> 00:14:28,495 think that the physical infrastructure out of band 397 00:14:28,495 --> 00:14:30,495 is actually better? I I I would prefer 398 00:14:30,495 --> 00:14:32,495 the physical. A mix of both, you know, 399 00:14:32,495 --> 00:14:34,355 for purposes. So you, 400 00:14:35,250 --> 00:14:37,889 here's how I would put it. If I 401 00:14:37,889 --> 00:14:40,309 would do priority one with the physical 402 00:14:40,769 --> 00:14:43,350 out of band infrastructure and then for failover 403 00:14:43,410 --> 00:14:46,210 or backup or purposes, you know, we we 404 00:14:46,210 --> 00:14:48,529 can never not have too much You can 405 00:14:48,529 --> 00:14:51,514 always just keep having Right? So you could 406 00:14:51,514 --> 00:14:54,735 have some kind of cloud based virtual VPN 407 00:14:55,115 --> 00:14:56,414 slash slash proxying 408 00:14:56,954 --> 00:14:59,115 approach to it, but I would still prefer 409 00:14:59,115 --> 00:15:01,934 for for us to have a physical infrastructure 410 00:15:02,074 --> 00:15:02,815 for OOB, 411 00:15:03,659 --> 00:15:05,100 as the first step. And then we can, 412 00:15:05,100 --> 00:15:06,320 you know, have virtual 413 00:15:06,860 --> 00:15:09,740 implementations to it. Even if it's a cloud 414 00:15:09,740 --> 00:15:12,639 connected, I would say. Like, even if it's 415 00:15:13,100 --> 00:15:15,519 via a cloud but is a separate physical 416 00:15:16,139 --> 00:15:16,639 infrastructure, 417 00:15:17,534 --> 00:15:19,934 I actually prefer the separate infrastructure myself as 418 00:15:19,934 --> 00:15:22,434 well. Tom, any thoughts on that particular question? 419 00:15:23,134 --> 00:15:24,735 Yeah. Yeah. I think you you want it 420 00:15:24,735 --> 00:15:27,375 to run independent of the of the, 421 00:15:28,095 --> 00:15:29,695 the the revenue plane, if you will, the 422 00:15:29,695 --> 00:15:32,014 data plane that carries the operations of your 423 00:15:32,014 --> 00:15:32,514 business. 424 00:15:33,836 --> 00:15:35,543 Yeah. I think that's 425 00:15:35,970 --> 00:15:38,769 running them separate is I think I would 426 00:15:38,769 --> 00:15:40,309 assume that's pretty well settled. 427 00:15:41,009 --> 00:15:42,769 You know, if you you don't really you 428 00:15:42,769 --> 00:15:44,370 don't have an out of band. It's not 429 00:15:44,370 --> 00:15:46,529 out of band if it's not separate, right? 430 00:15:46,529 --> 00:15:48,069 It's in band if it's separate. 431 00:15:48,455 --> 00:15:49,975 And so that's kind of to me, that 432 00:15:49,975 --> 00:15:52,075 it's the definition of out of band is 433 00:15:52,215 --> 00:15:53,115 separate physical, 434 00:15:54,294 --> 00:15:55,674 physical transmission medium. 435 00:15:56,054 --> 00:15:57,815 Interesting. So now if I were going to 436 00:15:57,815 --> 00:15:59,894 my manager to justify, you can you can 437 00:15:59,894 --> 00:16:01,815 critique my thoughts on this if you want 438 00:16:01,815 --> 00:16:02,315 to. 439 00:16:02,919 --> 00:16:04,460 It is this, that 440 00:16:05,799 --> 00:16:07,559 just like in routers and a lot of 441 00:16:07,559 --> 00:16:08,700 people don't know this. 442 00:16:09,399 --> 00:16:11,799 But when you build a router, when you 443 00:16:11,799 --> 00:16:14,059 are designing a router, boards, 444 00:16:14,440 --> 00:16:15,980 chipsets, everything else, 445 00:16:17,285 --> 00:16:19,365 In commercial grade gear and, by the way, 446 00:16:19,365 --> 00:16:21,684 this is something that's different about Linux based 447 00:16:21,684 --> 00:16:22,184 routers 448 00:16:22,565 --> 00:16:23,305 than about, 449 00:16:23,925 --> 00:16:25,305 like, Juniper, Cisco, 450 00:16:25,764 --> 00:16:26,264 whatever. 451 00:16:27,684 --> 00:16:29,365 There is actually a separate 452 00:16:29,925 --> 00:16:33,070 often, there is actually a separate internal channel 453 00:16:33,769 --> 00:16:34,509 for telemetry. 454 00:16:36,009 --> 00:16:36,509 Because 455 00:16:37,289 --> 00:16:39,629 it's it's an advertising it's a marketing thing. 456 00:16:39,850 --> 00:16:43,789 You can't say, oh, my router can support 457 00:16:44,125 --> 00:16:45,425 x gig per second 458 00:16:45,805 --> 00:16:46,785 if I'm producing 459 00:16:47,485 --> 00:16:49,585 x meg per second of telemetry 460 00:16:50,045 --> 00:16:51,904 and I'm sucking it off my primary 461 00:16:52,605 --> 00:16:55,345 backplane or network interface interconnect. 462 00:16:56,379 --> 00:16:57,980 Now this is more true. By the way, 463 00:16:57,980 --> 00:17:00,079 this is not necessarily true of 464 00:17:00,379 --> 00:17:02,699 single box, single FEs. But when you get 465 00:17:02,699 --> 00:17:04,480 into chassis boxes in particular, 466 00:17:05,179 --> 00:17:07,500 almost every chassis box I've ever worked on 467 00:17:07,500 --> 00:17:09,359 has a set of runs on the backplane 468 00:17:09,660 --> 00:17:10,960 that are nothing but telemetry. 469 00:17:11,605 --> 00:17:13,444 They're they're they're not there. They don't run 470 00:17:13,444 --> 00:17:15,284 customer traffic, or they don't they don't run 471 00:17:15,284 --> 00:17:17,765 through traffic. They only run internal into the 472 00:17:17,765 --> 00:17:18,984 inside the box traffic. 473 00:17:20,484 --> 00:17:22,484 Now if you think about it in those 474 00:17:22,484 --> 00:17:23,544 terms, you realize 475 00:17:23,845 --> 00:17:25,464 that if you are running 476 00:17:25,924 --> 00:17:26,424 your 477 00:17:28,250 --> 00:17:28,750 telemetry 478 00:17:29,210 --> 00:17:31,529 over the same network as you're carrying your 479 00:17:31,529 --> 00:17:32,349 data plane, 480 00:17:33,450 --> 00:17:35,950 you're doing exactly the same thing. 481 00:17:36,250 --> 00:17:37,150 You are consuming 482 00:17:37,690 --> 00:17:38,190 bandwidth. 483 00:17:38,730 --> 00:17:40,970 And maybe your bandwidth on your links is 484 00:17:40,970 --> 00:17:43,005 very, very low, And you're like, I don't 485 00:17:43,005 --> 00:17:44,845 really care. I only run 3%. 486 00:17:44,845 --> 00:17:46,384 I only run 5%. 487 00:17:47,085 --> 00:17:47,585 But 488 00:17:48,125 --> 00:17:50,625 there's two problems with that. The first is, 489 00:17:51,884 --> 00:17:54,065 there's always the issue of 490 00:17:55,670 --> 00:17:56,170 measuring 491 00:17:56,710 --> 00:17:58,490 things, changing things, 492 00:17:59,430 --> 00:18:01,109 which is something we don't think about very 493 00:18:01,109 --> 00:18:02,009 much in networking. 494 00:18:02,390 --> 00:18:03,529 But it's true. 495 00:18:03,990 --> 00:18:06,329 When you measure things, you change them. 496 00:18:06,789 --> 00:18:09,049 And the more you can separate the measurement 497 00:18:09,704 --> 00:18:11,005 from what you're measuring, 498 00:18:11,625 --> 00:18:13,804 the less you're going to change things. 499 00:18:14,585 --> 00:18:16,184 So this is just a, 500 00:18:17,304 --> 00:18:19,944 this is just the way that things work 501 00:18:19,944 --> 00:18:20,924 in the real world. 502 00:18:21,304 --> 00:18:22,825 And the other thing is, again, when you 503 00:18:22,825 --> 00:18:24,044 come down to an emergency 504 00:18:24,580 --> 00:18:26,740 or you have massive problems, like, if I 505 00:18:26,740 --> 00:18:27,960 have a routing loop 506 00:18:28,740 --> 00:18:31,160 that's consuming 80% of my link, 507 00:18:31,779 --> 00:18:32,279 it's 508 00:18:32,740 --> 00:18:33,240 stopping 509 00:18:33,940 --> 00:18:34,680 my telemetry 510 00:18:34,980 --> 00:18:37,160 traffic from making it to my system 511 00:18:37,539 --> 00:18:38,359 that discovers 512 00:18:38,660 --> 00:18:39,960 the routing loop. 513 00:18:40,654 --> 00:18:41,154 Okay? 514 00:18:41,934 --> 00:18:42,835 Like, stop that. 515 00:18:45,534 --> 00:18:46,515 Don't do that. 516 00:18:47,214 --> 00:18:48,815 So to me, when I build the out 517 00:18:48,815 --> 00:18:50,974 of van network, whether it's proxy or VPN 518 00:18:50,974 --> 00:18:52,515 or whether it's physical infrastructure, 519 00:18:53,200 --> 00:18:55,200 I'm gonna put all of my telemetry on 520 00:18:55,200 --> 00:18:55,859 that network. 521 00:18:56,799 --> 00:18:58,720 Not just remote. I'm gonna put everything I 522 00:18:58,720 --> 00:19:00,420 can on that network that is 523 00:19:00,720 --> 00:19:02,960 what you know, whatever it is. So I'm 524 00:19:02,960 --> 00:19:05,359 gonna put my network management system and my 525 00:19:05,359 --> 00:19:06,420 automation system 526 00:19:06,799 --> 00:19:08,259 on that secondary network 527 00:19:08,644 --> 00:19:09,704 and never run 528 00:19:10,085 --> 00:19:12,184 it. I'm gonna have a clear, clean separation. 529 00:19:12,964 --> 00:19:14,105 So I don't know. 530 00:19:14,644 --> 00:19:16,484 Maybe maybe I'm a little bit sensitive to 531 00:19:16,484 --> 00:19:19,204 the measurement changes things, and I've seen cases 532 00:19:19,204 --> 00:19:19,704 where 533 00:19:21,329 --> 00:19:22,710 I was working with a bank, 534 00:19:23,250 --> 00:19:25,890 and they were getting eEdge RP SIAs, which 535 00:19:25,890 --> 00:19:27,750 tells you how long ago this was. 536 00:19:28,130 --> 00:19:28,630 And 537 00:19:29,650 --> 00:19:30,390 we put 538 00:19:31,329 --> 00:19:31,829 packet, 539 00:19:32,450 --> 00:19:34,390 we put packet dumps on all the interfaces 540 00:19:34,835 --> 00:19:36,035 to try to figure out what was going 541 00:19:36,035 --> 00:19:37,954 on because we couldn't figure out from the 542 00:19:37,954 --> 00:19:40,194 EIGRP perspective, like, why we were getting stuck 543 00:19:40,194 --> 00:19:40,775 in actives. 544 00:19:41,634 --> 00:19:43,494 And it turned out 545 00:19:43,875 --> 00:19:46,035 that every time we did a packet dump, 546 00:19:46,035 --> 00:19:47,335 because we saw an SAA, 547 00:19:48,099 --> 00:19:50,659 all the traffic in the interface queue was 548 00:19:50,659 --> 00:19:51,639 management traffic, 549 00:19:53,700 --> 00:19:54,200 which 550 00:19:54,740 --> 00:19:57,960 made it impossible to troubleshoot the actual problem. 551 00:19:58,819 --> 00:19:59,319 Right? 552 00:19:59,700 --> 00:20:01,559 So sometimes you just gotta, like, 553 00:20:01,940 --> 00:20:02,440 stop, 554 00:20:02,825 --> 00:20:04,845 Take this stuff out. Make it separate 555 00:20:05,384 --> 00:20:07,704 because it makes it easier to troubleshoot. It 556 00:20:07,704 --> 00:20:09,884 makes it easier to see what's going on. 557 00:20:10,105 --> 00:20:12,664 So, Adam, thoughts on that at all? Daryl, 558 00:20:12,664 --> 00:20:14,365 do you have start with you. 559 00:20:16,190 --> 00:20:17,650 I I agree with that viewpoint, 560 00:20:17,950 --> 00:20:19,089 Ross. I I think, 561 00:20:20,190 --> 00:20:23,890 in general, when we deal with technical systems, 562 00:20:24,829 --> 00:20:26,750 complex systems, as a matter of fact, right? 563 00:20:26,750 --> 00:20:29,230 These are very complex systems in today's today's, 564 00:20:29,470 --> 00:20:30,369 day and age. 565 00:20:30,765 --> 00:20:32,065 I think separation 566 00:20:32,845 --> 00:20:34,144 as much as you can 567 00:20:34,684 --> 00:20:36,285 is probably a good thing. You know, 568 00:20:37,244 --> 00:20:38,305 there there was, 569 00:20:39,404 --> 00:20:41,244 I I I there was a paper back 570 00:20:41,244 --> 00:20:42,924 in college, that we had. It was a 571 00:20:42,924 --> 00:20:45,164 software engineering related paper, and there was a 572 00:20:45,164 --> 00:20:47,380 line that stuck with me. It it wrote 573 00:20:47,380 --> 00:20:50,019 something like like this. It said, the the 574 00:20:50,019 --> 00:20:50,759 best approach 575 00:20:51,220 --> 00:20:53,700 to writing code and and software engineering would 576 00:20:53,700 --> 00:20:54,599 be to have 577 00:20:55,140 --> 00:20:57,240 high cohesion and low coupling. 578 00:20:57,619 --> 00:20:59,619 So I would really prefer for the network 579 00:20:59,619 --> 00:21:01,460 to have the same kind of principle where 580 00:21:01,460 --> 00:21:02,039 the coupling 581 00:21:02,505 --> 00:21:04,025 is minimized as much as you can, and 582 00:21:04,025 --> 00:21:06,204 then you can increase the cohe cohesiveness 583 00:21:06,825 --> 00:21:08,265 as much as you can and as much 584 00:21:08,265 --> 00:21:10,904 as you want. But I think coupling of 585 00:21:10,904 --> 00:21:11,724 these components, 586 00:21:12,825 --> 00:21:15,785 whether there's management or or control plane or 587 00:21:15,785 --> 00:21:18,179 routing protocols, etcetera, etcetera, I don't think that's 588 00:21:18,179 --> 00:21:20,740 a good idea because of exactly the kind 589 00:21:20,740 --> 00:21:23,319 of examples you shared with us. I think 590 00:21:23,940 --> 00:21:26,419 low coupling and high cohesion should be applied 591 00:21:26,419 --> 00:21:29,139 to network engineering in general, not just in 592 00:21:29,139 --> 00:21:30,819 software engineering. I think we need to apply 593 00:21:30,819 --> 00:21:33,964 that to, networks as well. So I completely 594 00:21:34,025 --> 00:21:35,724 agree with you, Ross, on that. Yeah. 595 00:23:53,869 --> 00:23:55,789 Yeah. And and I think the first counter 596 00:23:55,789 --> 00:23:58,289 to that would be, is there any historical 597 00:23:58,429 --> 00:23:58,929 data 598 00:23:59,470 --> 00:23:59,970 about 599 00:24:01,470 --> 00:24:03,470 and and there may not be any historical 600 00:24:03,470 --> 00:24:05,924 cases in a particular case where they've been 601 00:24:05,924 --> 00:24:07,605 locked out of a router and had to 602 00:24:07,605 --> 00:24:08,825 ship somebody on-site. 603 00:24:09,924 --> 00:24:10,825 And and 604 00:24:11,524 --> 00:24:12,664 but if there are, 605 00:24:13,125 --> 00:24:14,585 how much did that cost? 606 00:24:15,444 --> 00:24:17,625 And and what what does the cost differential? 607 00:24:17,924 --> 00:24:19,304 If I lose a store 608 00:24:20,039 --> 00:24:21,180 for a week 609 00:24:22,039 --> 00:24:24,539 and I cannot do any access to it, 610 00:24:25,240 --> 00:24:27,400 what am is that store gonna stay open 611 00:24:27,400 --> 00:24:29,019 if my margins are that tight? 612 00:24:29,880 --> 00:24:30,380 Right? 613 00:24:30,840 --> 00:24:33,320 So there's always like this so I worked 614 00:24:33,320 --> 00:24:35,080 on one network once, which was a retail 615 00:24:35,080 --> 00:24:37,305 network, where they solve this problem a little 616 00:24:37,305 --> 00:24:39,325 bit, like, perhaps oddly, 617 00:24:39,625 --> 00:24:41,944 is they actually had three different connections into 618 00:24:41,944 --> 00:24:43,164 every store, physically. 619 00:24:45,065 --> 00:24:47,164 They had some type of a local provider, 620 00:24:48,119 --> 00:24:50,200 whatever it was, at the time, you know, 621 00:24:50,200 --> 00:24:51,880 d s three, whatever it was, it happened 622 00:24:51,880 --> 00:24:52,539 to be. 623 00:24:53,240 --> 00:24:53,740 Then 624 00:24:54,119 --> 00:24:56,220 they found a local cable 625 00:24:56,519 --> 00:24:57,019 provider 626 00:24:58,279 --> 00:25:00,599 that was completely diverse, and they didn't buy 627 00:25:00,599 --> 00:25:03,335 a business level plan. They just bought, like, 628 00:25:03,335 --> 00:25:04,715 your bog standard 629 00:25:05,255 --> 00:25:07,914 $20 a month, send me your modem, 630 00:25:08,375 --> 00:25:09,914 give me an Ethernet connection, 631 00:25:10,535 --> 00:25:12,134 and I'm gonna stick it on the back 632 00:25:12,134 --> 00:25:13,275 of my store router. 633 00:25:14,055 --> 00:25:16,075 And then they had a satellite connection. 634 00:25:17,015 --> 00:25:19,330 All three. Now, perhaps, that's overkill. 635 00:25:20,029 --> 00:25:22,109 But what they found was was that to 636 00:25:22,109 --> 00:25:23,950 justify it, the way they did it was 637 00:25:23,950 --> 00:25:24,609 they said, 638 00:25:25,150 --> 00:25:27,549 we are going to have something in the 639 00:25:27,549 --> 00:25:28,049 router 640 00:25:28,670 --> 00:25:31,009 that if the store loses connectivity, 641 00:25:31,734 --> 00:25:34,214 we're gonna ship all of the credit card 642 00:25:34,214 --> 00:25:35,835 information over the 643 00:25:36,855 --> 00:25:37,674 cable modem 644 00:25:38,774 --> 00:25:41,034 so we can continue doing business. 645 00:25:41,575 --> 00:25:42,954 We may not have inventory. 646 00:25:43,255 --> 00:25:45,255 We may not have video. We may not 647 00:25:45,255 --> 00:25:47,799 have video cameras. We may not have anything 648 00:25:47,799 --> 00:25:50,940 else, but we're going to process credit cards. 649 00:25:51,480 --> 00:25:53,080 And we're going to have two ways of 650 00:25:53,080 --> 00:25:56,039 processing credit cards. So this store will stay 651 00:25:56,039 --> 00:25:56,539 open 652 00:25:57,320 --> 00:25:59,080 all the time. There will not be a 653 00:25:59,080 --> 00:26:01,875 day when this store is closed unless there's, 654 00:26:01,875 --> 00:26:02,934 like, a hurricane, 655 00:26:03,315 --> 00:26:05,335 you know, and it takes out all connectivity 656 00:26:05,474 --> 00:26:06,134 and power. 657 00:26:06,914 --> 00:26:09,715 But if the primary provider goes down, well, 658 00:26:09,715 --> 00:26:11,474 you know, we'll lose video feed from the 659 00:26:11,474 --> 00:26:11,974 store. 660 00:26:12,515 --> 00:26:14,930 Oh, well. But we won't lose credit card 661 00:26:14,930 --> 00:26:15,430 transactions. 662 00:26:16,130 --> 00:26:18,130 And so that's the way they justified it, 663 00:26:18,130 --> 00:26:19,029 was they said, 664 00:26:20,130 --> 00:26:23,170 this increases your revenue over time because you 665 00:26:23,170 --> 00:26:24,849 can look back at how many times that 666 00:26:24,849 --> 00:26:27,105 primary link has gone down during the year. 667 00:26:27,265 --> 00:26:29,265 And even if it's for two or three 668 00:26:29,265 --> 00:26:31,845 hours that they can't process credit cards, 669 00:26:33,265 --> 00:26:35,825 then, you know, you're in a completely different 670 00:26:35,825 --> 00:26:36,325 situation 671 00:26:36,865 --> 00:26:38,704 when you can justify the cost of that 672 00:26:38,704 --> 00:26:42,164 second link just on normal operational procedures. 673 00:26:42,799 --> 00:26:43,700 And then that can 674 00:26:47,359 --> 00:26:48,109 become your 675 00:28:09,865 --> 00:28:11,005 Yeah. Right. 676 00:28:11,384 --> 00:28:14,125 Right. Yeah. ATM machines are the same way. 677 00:28:14,345 --> 00:28:16,444 A lot of ATM machines are double connected. 678 00:28:17,144 --> 00:28:18,744 Not not all of them. That used to 679 00:28:18,744 --> 00:28:21,144 be more common, I believe. Maybe it's not 680 00:28:21,144 --> 00:28:22,044 as much anymore. 681 00:28:22,460 --> 00:28:23,980 But there always used to be a separate 682 00:28:23,980 --> 00:28:26,220 connection from the ATM machine to the central 683 00:28:26,220 --> 00:28:26,720 bank 684 00:28:27,099 --> 00:28:27,599 office 685 00:28:27,980 --> 00:28:30,319 and the connection through the local office. 686 00:28:30,700 --> 00:28:32,640 And that was for this very same reason. 687 00:28:33,180 --> 00:28:35,740 The bank loses revenue when the ATM machine 688 00:28:35,740 --> 00:28:36,525 doesn't work. 689 00:28:37,005 --> 00:28:38,845 And a customer comes by and the ATM 690 00:28:38,845 --> 00:28:39,984 machine doesn't work, 691 00:28:40,285 --> 00:28:42,684 they're gonna remember that in a month, and 692 00:28:42,684 --> 00:28:45,025 they're gonna go find a different ATM machine. 693 00:28:45,565 --> 00:28:48,205 Right? And so it's it's not just revenue, 694 00:28:48,205 --> 00:28:49,025 it's mindshare. 695 00:28:50,140 --> 00:28:52,460 Right? And and so we we we do, 696 00:28:52,460 --> 00:28:53,819 Tom, like you said, we do need to 697 00:28:53,819 --> 00:28:54,960 think in these terms, 698 00:28:55,819 --> 00:28:58,380 creative ways to create an out of band 699 00:28:58,380 --> 00:29:00,079 network and justify it. 700 00:29:01,099 --> 00:29:03,455 So that's that's kind of that's a that's 701 00:29:03,455 --> 00:29:05,455 a thing with me. So, Daryl, I don't 702 00:29:05,455 --> 00:29:07,055 know. You haven't said anything in a while. 703 00:29:07,055 --> 00:29:08,434 Tom and I have been, like 704 00:29:10,095 --> 00:29:13,234 No. You did mention something interesting earlier. So, 705 00:29:13,775 --> 00:29:15,869 you said you you mentioned there was a 706 00:29:15,869 --> 00:29:18,430 business that actually made use of satellite for 707 00:29:18,430 --> 00:29:18,589 the, 708 00:29:19,549 --> 00:29:22,829 you know, OOB access. Right? So interestingly, I've 709 00:29:22,829 --> 00:29:25,150 done the same thing except with Starlink. So 710 00:29:25,150 --> 00:29:25,950 there was this, 711 00:29:26,589 --> 00:29:27,089 wireless 712 00:29:27,390 --> 00:29:30,154 ISP project I did. I believe they were 713 00:29:30,154 --> 00:29:32,255 based in Montana up in the mountains. 714 00:29:32,714 --> 00:29:34,795 And so as you can probably imagine, there's 715 00:29:34,795 --> 00:29:36,634 not a lot of fiber connectivity in in 716 00:29:36,634 --> 00:29:37,454 that particular 717 00:29:37,835 --> 00:29:40,734 area slash region that they were operating in. 718 00:29:41,595 --> 00:29:42,095 So 719 00:29:42,569 --> 00:29:43,069 I 720 00:29:43,369 --> 00:29:45,450 got the idea that, hey. Why don't I 721 00:29:45,450 --> 00:29:47,950 just use since they already had Starlink connectivity, 722 00:29:48,409 --> 00:29:50,169 why don't we just put a Starlink up 723 00:29:50,169 --> 00:29:51,769 in in this pop, in this, 724 00:29:52,329 --> 00:29:54,730 in this site and use I p v 725 00:29:54,730 --> 00:29:55,869 six from Starlink 726 00:29:56,575 --> 00:29:59,295 for the OB access. So that was it. 727 00:29:59,295 --> 00:30:00,815 Like, you know, I could any of us 728 00:30:00,815 --> 00:30:03,075 from anywhere on a planet could then remotely 729 00:30:03,215 --> 00:30:04,035 access those 730 00:30:04,414 --> 00:30:07,134 devices on that site up in, up in 731 00:30:07,134 --> 00:30:07,795 the mountains 732 00:30:08,255 --> 00:30:10,815 over I p v six, over Starling without 733 00:30:10,815 --> 00:30:12,015 having to worry about, 734 00:30:12,599 --> 00:30:14,519 you know, CGNET and all all those kind 735 00:30:14,519 --> 00:30:16,359 of stuff, you know, like a reverse proxy 736 00:30:16,359 --> 00:30:17,579 and all that kind of stuff. 737 00:30:18,119 --> 00:30:18,359 So, 738 00:30:19,400 --> 00:30:21,160 this is also another key point that, 739 00:30:22,039 --> 00:30:24,679 for any OOB project that I do, IP 740 00:30:24,679 --> 00:30:26,359 v six is a must have because, like, 741 00:30:26,359 --> 00:30:28,335 I would like everything to be routed. I'm 742 00:30:28,335 --> 00:30:29,534 not a fan of NAT. I'm not a 743 00:30:29,534 --> 00:30:31,934 fan of all these layers of, you know, 744 00:30:31,934 --> 00:30:34,734 proxying and and NAT ing and triple net. 745 00:30:34,734 --> 00:30:37,054 I've even seen triple net before. I'm sure 746 00:30:37,054 --> 00:30:38,974 some of us have seen triple net. I 747 00:30:38,974 --> 00:30:40,654 I I'm I'm I'm just wondering connection is 748 00:30:40,654 --> 00:30:41,474 triple netted. 749 00:30:42,619 --> 00:30:45,019 I'll bet I'll just about bet everybody's home 750 00:30:45,019 --> 00:30:47,200 connection is triple netted. So sorry. Go ahead. 751 00:30:48,460 --> 00:30:49,200 Yeah. Yeah. 752 00:30:49,740 --> 00:30:51,980 So, yeah, I I I think IP v 753 00:30:51,980 --> 00:30:53,899 six is also very important for the OOB 754 00:30:53,899 --> 00:30:56,494 because it's also for simplicity. Right? If everything 755 00:30:56,494 --> 00:30:58,174 is just routed, like, you don't have to 756 00:30:58,174 --> 00:31:00,335 worry about all kinds of these problems with 757 00:31:00,335 --> 00:31:02,255 remote access. If everything has a goa on 758 00:31:02,255 --> 00:31:04,095 it, of course, you're gonna have a stateful 759 00:31:04,095 --> 00:31:06,674 firewall protecting those devices behind, 760 00:31:07,190 --> 00:31:09,269 the OOB network. You know, from from a 761 00:31:09,269 --> 00:31:11,849 security standpoint, you can have s, SPI 762 00:31:12,390 --> 00:31:15,190 in front, in in in between those devices 763 00:31:15,190 --> 00:31:17,589 and the edge of the OOB network, protecting 764 00:31:17,589 --> 00:31:20,390 them from, from the public Internet with a 765 00:31:20,390 --> 00:31:22,309 public IP v six address. But I think 766 00:31:22,309 --> 00:31:24,254 that was that's that's an important, 767 00:31:24,654 --> 00:31:26,174 attribute to think about. And as far as 768 00:31:26,174 --> 00:31:29,154 creativity goes with OOB, I think Starlink 769 00:31:29,855 --> 00:31:31,934 I think we'll probably see more people making 770 00:31:31,934 --> 00:31:34,754 good use of Starlink for business applications, including 771 00:31:34,974 --> 00:31:35,954 out of band network. 772 00:31:36,414 --> 00:31:36,994 I think 773 00:31:37,440 --> 00:31:38,740 styling is a very interesting, 774 00:31:39,679 --> 00:31:42,179 tool in the toolbox for for network engineers 775 00:31:42,240 --> 00:31:43,859 going forward is what I think. 776 00:31:44,240 --> 00:31:46,480 And now the other interesting bit that, 777 00:31:47,039 --> 00:31:48,339 Tom mentioned was, 778 00:31:48,960 --> 00:31:50,720 you know, a lot of network engineers are 779 00:31:50,720 --> 00:31:52,819 waiting on the finance 780 00:31:53,144 --> 00:31:56,184 guys or finance team of their employers or 781 00:31:56,184 --> 00:31:56,684 company, 782 00:31:57,384 --> 00:31:59,785 to try and address some of these, concerns 783 00:31:59,785 --> 00:32:01,085 or issues we have. 784 00:32:01,625 --> 00:32:02,125 And, 785 00:32:02,505 --> 00:32:04,424 it it does kind of, like, debate from 786 00:32:04,424 --> 00:32:06,184 the original topic here, which is out of 787 00:32:06,184 --> 00:32:08,070 band networks. But I I 788 00:32:08,450 --> 00:32:10,470 I think in in this day and age, 789 00:32:10,609 --> 00:32:13,029 it's very difficult to just be to just 790 00:32:13,250 --> 00:32:15,430 quote, unquote, just be an engineer 791 00:32:15,970 --> 00:32:18,710 without learning some amount of finance and accounting 792 00:32:18,769 --> 00:32:20,690 and some of the business side of things. 793 00:32:20,690 --> 00:32:21,590 Right? I think 794 00:32:21,924 --> 00:32:24,964 I think we we we as as humans, 795 00:32:24,964 --> 00:32:26,884 we as a civilization, I think we have 796 00:32:26,884 --> 00:32:28,025 grown very complex, 797 00:32:28,884 --> 00:32:30,644 and we have all kinds of, 798 00:32:31,684 --> 00:32:34,325 of facets to our very fabric of the 799 00:32:34,325 --> 00:32:36,029 society whereby you can't just 800 00:32:36,509 --> 00:32:37,730 be a quote, unquote, 801 00:32:38,349 --> 00:32:40,509 network engineer. I think you need to learn 802 00:32:40,509 --> 00:32:42,190 quite a bit more than that to actually 803 00:32:42,190 --> 00:32:43,869 be able to survive in that given role, 804 00:32:43,869 --> 00:32:46,190 and this is not just for network guys. 805 00:32:46,349 --> 00:32:48,430 You know, this applies applies to any kind 806 00:32:48,430 --> 00:32:49,890 of job out there in the software, 807 00:32:50,434 --> 00:32:52,134 or whatever it may be, I think. 808 00:32:52,595 --> 00:32:54,755 But but it also does create some layer 809 00:32:54,755 --> 00:32:56,515 eight challenges as I like to call it 810 00:32:56,515 --> 00:32:57,015 because 811 00:32:57,394 --> 00:32:58,994 in many of these cases, and this is 812 00:32:58,994 --> 00:33:00,914 from my own conversations with a lot of 813 00:33:00,914 --> 00:33:02,855 our fellow professionals in the industry, 814 00:33:03,759 --> 00:33:05,940 the ground reality is oftentimes 815 00:33:06,319 --> 00:33:09,440 network network engineers or network staff are looked 816 00:33:09,440 --> 00:33:12,019 upon as a call center by the stakeholders 817 00:33:12,160 --> 00:33:12,819 of a business. 818 00:33:13,440 --> 00:33:15,759 And what that means, the bottom line is 819 00:33:15,759 --> 00:33:18,079 a lot of these engineers are underpaid. They're 820 00:33:18,079 --> 00:33:21,144 they're not really paid what their expertise brings 821 00:33:21,144 --> 00:33:22,744 in or what value they deliver to the 822 00:33:22,744 --> 00:33:25,384 company. They're underpaid, so which then leads into 823 00:33:25,384 --> 00:33:26,365 this whole cyclic, 824 00:33:26,904 --> 00:33:28,525 dependency problem whereby 825 00:33:28,904 --> 00:33:30,505 they're not paid enough. They know what the 826 00:33:30,505 --> 00:33:32,184 problem is. They know what the solution is. 827 00:33:32,184 --> 00:33:33,785 But, again, they're paid enough so they don't 828 00:33:33,785 --> 00:33:35,990 bring these these concerns to the leadership 829 00:33:36,529 --> 00:33:39,250 or finance department of the company. So then 830 00:33:39,250 --> 00:33:41,250 the problem never gets solved. They leave the 831 00:33:41,250 --> 00:33:43,890 company. Someone else comes in. Same problem. They 832 00:33:43,890 --> 00:33:45,509 leave. They go. There's a lot of churn 833 00:33:45,570 --> 00:33:47,809 in that ways, and nothing gets resolved at, 834 00:33:48,210 --> 00:33:50,070 for ten, fifteen years down the road. 835 00:33:50,585 --> 00:33:52,505 So I I I don't know what would 836 00:33:52,505 --> 00:33:53,325 be the right, 837 00:33:54,265 --> 00:33:56,424 way to tackle this issue, but, 838 00:33:56,904 --> 00:33:58,585 but it's definitely an issue out there. It's 839 00:33:58,585 --> 00:33:59,785 a little it's a layer e issue. It's 840 00:33:59,785 --> 00:34:02,045 less of a engineering problem and more of 841 00:34:02,105 --> 00:34:03,704 a human issue, I think. You know, a 842 00:34:03,704 --> 00:34:05,650 layer eight issue, maybe even nine and ten 843 00:34:05,650 --> 00:34:06,549 in some cases. 844 00:34:08,449 --> 00:34:10,369 So, yeah, maybe, you guys have some thoughts 845 00:34:10,369 --> 00:34:13,170 on that. So I completely agree. We've really 846 00:34:13,170 --> 00:34:15,429 bought too much the idea of 847 00:34:16,050 --> 00:34:18,449 jack of all trades, master of none. I 848 00:34:18,449 --> 00:34:20,309 hate that saying. I think that people 849 00:34:21,094 --> 00:34:21,914 really overplay 850 00:34:22,694 --> 00:34:23,194 it. 851 00:34:23,654 --> 00:34:25,414 And, you know, oh, you think you know 852 00:34:25,414 --> 00:34:26,934 about philosophy and you think you know about 853 00:34:26,934 --> 00:34:28,614 networking. You think, oh, you can't you can't 854 00:34:28,614 --> 00:34:29,594 know all that stuff. 855 00:34:30,614 --> 00:34:31,755 No. That's wrong. 856 00:34:32,214 --> 00:34:34,980 Like, you as a human should be constantly 857 00:34:35,119 --> 00:34:36,820 expanding the scope of your knowledge 858 00:34:37,119 --> 00:34:38,880 and and the mental maps you have of 859 00:34:38,880 --> 00:34:39,460 the world 860 00:34:39,840 --> 00:34:41,539 in useful and helpful ways. 861 00:34:42,079 --> 00:34:43,700 And that's just and, of course, 862 00:34:44,160 --> 00:34:46,400 the younger you are, and this is not 863 00:34:46,400 --> 00:34:48,160 to say anything bad against young people or 864 00:34:48,160 --> 00:34:51,054 anything, It's just a life experience thing. 865 00:34:51,434 --> 00:34:51,934 It's 866 00:34:52,474 --> 00:34:54,635 you know more after you've been doing something 867 00:34:54,635 --> 00:34:56,954 for twenty years and dealing with a lot 868 00:34:56,954 --> 00:34:57,934 of different businesses 869 00:34:58,315 --> 00:35:00,315 than you do when you've only been dealing 870 00:35:00,315 --> 00:35:02,349 with it for five years. And you might 871 00:35:02,349 --> 00:35:04,190 think, oh, five years is enough. I know 872 00:35:04,190 --> 00:35:05,889 everything. No. Trust me. 873 00:35:06,190 --> 00:35:08,750 You'll learn more. You should be learning more. 874 00:35:08,750 --> 00:35:11,250 It should be your goal to learn more. 875 00:35:11,710 --> 00:35:14,349 And so that is that is something that 876 00:35:14,349 --> 00:35:16,369 I don't think we take seriously enough 877 00:35:16,784 --> 00:35:17,284 in 878 00:35:18,224 --> 00:35:18,885 our world, 879 00:35:19,424 --> 00:35:20,964 in technology in particular. 880 00:35:21,344 --> 00:35:23,444 We tend to idolize people who know 881 00:35:23,744 --> 00:35:24,885 one or two things. 882 00:35:25,505 --> 00:35:27,505 Like, we we we really idolize the guy 883 00:35:27,505 --> 00:35:29,585 who's a c coder, who knows every in 884 00:35:29,585 --> 00:35:31,849 and out of and can tell you, you 885 00:35:31,849 --> 00:35:34,250 know, oh my goodness. That's c 19 80 886 00:35:34,250 --> 00:35:36,329 four, and that's c nineteen eighty six, and 887 00:35:36,329 --> 00:35:38,010 that was put into the spec on this 888 00:35:38,010 --> 00:35:40,250 date and by so and so for we 889 00:35:40,250 --> 00:35:41,469 really idolize that. 890 00:35:41,849 --> 00:35:44,505 That's really that's the those are really cool 891 00:35:44,505 --> 00:35:46,505 people to be around, but the honest truth 892 00:35:46,505 --> 00:35:48,204 is they're not as useful 893 00:35:48,824 --> 00:35:50,525 to the real world business 894 00:35:50,984 --> 00:35:52,284 as somebody who 895 00:35:52,824 --> 00:35:55,704 knows a broader range of things. So I 896 00:35:55,704 --> 00:35:58,429 think that's that's always true. And going back 897 00:35:58,429 --> 00:35:59,730 to out of band networks, 898 00:36:00,510 --> 00:36:02,429 I think knowing a lot of things helps 899 00:36:02,429 --> 00:36:05,069 you justify building a network that's better for 900 00:36:05,069 --> 00:36:05,730 the business. 901 00:36:06,670 --> 00:36:08,589 I think that's that's that's an important 902 00:36:09,565 --> 00:36:11,405 it's not just for me, it is for 903 00:36:11,405 --> 00:36:13,965 me, it's not just for me, it's also 904 00:36:13,965 --> 00:36:14,784 for the business. 905 00:36:15,244 --> 00:36:16,065 So I think 906 00:36:16,925 --> 00:36:17,664 those are very, 907 00:36:19,005 --> 00:36:20,784 important things to think about. 908 00:36:21,485 --> 00:36:23,405 So that's kinda my take on the whole 909 00:36:23,405 --> 00:36:25,349 range thing. It's been my take for a 910 00:36:25,349 --> 00:36:27,130 long time. There's a great book called Range. 911 00:36:27,429 --> 00:36:29,429 I I love that book. It's it's being 912 00:36:29,429 --> 00:36:31,670 being a fox in a hedgehog world, I 913 00:36:31,670 --> 00:36:33,769 think is the subtitle or something like that. 914 00:36:34,630 --> 00:36:36,789 Or the or the value of something like 915 00:36:36,789 --> 00:36:37,449 that. Anyway, 916 00:36:37,909 --> 00:36:38,409 the 917 00:36:38,735 --> 00:36:40,974 value of the generalist in a specialist world, 918 00:36:40,974 --> 00:36:42,974 something like that. So it's very I think 919 00:36:42,974 --> 00:36:44,815 it's a very important thing to to get 920 00:36:44,815 --> 00:36:46,355 into and understand. 921 00:36:48,175 --> 00:36:50,094 So back to out of band, how this 922 00:36:50,094 --> 00:36:51,695 all relates to that is, again, it helps 923 00:36:51,695 --> 00:36:52,914 you justify things. 924 00:36:53,390 --> 00:36:55,410 And it helps you build a better network. 925 00:36:56,030 --> 00:36:57,570 Starlink, I think, is interesting. 926 00:36:58,269 --> 00:37:00,190 Now, there is a new attack surface when 927 00:37:00,190 --> 00:37:01,869 you do this. Don't think that there's no 928 00:37:01,869 --> 00:37:02,369 tradeoffs. 929 00:37:04,110 --> 00:37:06,585 Right? This is something we often do. This 930 00:37:06,585 --> 00:37:08,344 is all beautiful. It's called build build out 931 00:37:08,344 --> 00:37:09,164 of band networks. 932 00:37:09,704 --> 00:37:11,545 You have to secure the out of band 933 00:37:11,545 --> 00:37:12,045 network. 934 00:37:13,065 --> 00:37:14,284 Like, it it's got 935 00:37:14,905 --> 00:37:18,264 it it needs SSH or it needs something. 936 00:37:18,264 --> 00:37:20,445 Whatever you normally do, it needs something. 937 00:37:21,000 --> 00:37:22,300 Don't don't assume 938 00:37:23,239 --> 00:37:25,400 that no one's gonna find the right IP 939 00:37:25,400 --> 00:37:25,900 address. 940 00:37:27,480 --> 00:37:29,960 Yeah. That's that's not gonna happen. So Well, 941 00:37:29,960 --> 00:37:32,300 and we're and we're and we're tempted, sometimes, 942 00:37:32,840 --> 00:37:34,519 building out of band networks to say, well, 943 00:37:34,519 --> 00:37:36,139 if we're gonna use this in an emergency, 944 00:37:36,454 --> 00:37:38,215 it needs to be really easy. It needs 945 00:37:38,215 --> 00:37:39,815 to be, like, I don't I shouldn't have 946 00:37:39,815 --> 00:37:42,054 to go through any any special hoops. I 947 00:37:42,054 --> 00:37:43,255 should because I'm gonna need to get in 948 00:37:43,255 --> 00:37:44,775 there fast. And you have to you have 949 00:37:44,775 --> 00:37:46,375 to fight that. You have to just it's 950 00:37:46,375 --> 00:37:48,215 it's another network that has to be secured 951 00:37:48,215 --> 00:37:50,215 like everything else. Yeah. Yeah. It's a part 952 00:37:50,215 --> 00:37:52,019 of your attack surface, and you should you 953 00:37:52,019 --> 00:37:52,760 should evaluate 954 00:37:53,539 --> 00:37:55,780 it in that way. So And and it 955 00:37:55,780 --> 00:37:56,280 quickly 956 00:37:56,739 --> 00:37:58,260 expands once you say, oh, this is a 957 00:37:58,260 --> 00:37:59,699 great idea. I'm gonna start doing it. All 958 00:37:59,699 --> 00:38:01,400 of a sudden, you'll be like, oh, 959 00:38:01,860 --> 00:38:03,300 how do I get how do I get 960 00:38:03,300 --> 00:38:05,795 my authentication systems under this segment? How do 961 00:38:05,795 --> 00:38:07,255 I do I have to put two interfaces 962 00:38:07,315 --> 00:38:07,894 in everything? 963 00:38:08,355 --> 00:38:09,875 Do I you know, it it starts it 964 00:38:09,954 --> 00:38:11,795 like, it's not a it's not just a 965 00:38:11,795 --> 00:38:12,675 matter of, 966 00:38:12,994 --> 00:38:14,755 throw in a, you know, a cheap cable 967 00:38:14,755 --> 00:38:15,255 modem 968 00:38:15,555 --> 00:38:17,875 and put a firewall on it, and we're 969 00:38:17,875 --> 00:38:19,730 good to go. You have to you have 970 00:38:19,730 --> 00:38:21,809 to think through it. It requires it requires 971 00:38:21,809 --> 00:38:23,670 design work. Yeah. You know, 972 00:38:24,050 --> 00:38:26,389 speaking about security for out of band networks, 973 00:38:26,929 --> 00:38:29,489 I I I think I like, the way 974 00:38:29,489 --> 00:38:31,349 the way I take network security, 975 00:38:32,355 --> 00:38:34,114 POV point of view, the way I, you 976 00:38:34,114 --> 00:38:36,695 know, the way I tackle network security, right, 977 00:38:37,715 --> 00:38:39,815 it ties back to my 978 00:38:40,275 --> 00:38:43,155 background where I started learning networking with with 979 00:38:43,155 --> 00:38:45,315 Linux based systems. That means Linux and the 980 00:38:45,315 --> 00:38:47,710 filter framework. So I'm pretty familiar with Linux 981 00:38:47,710 --> 00:38:48,690 Netfilter Framework. 982 00:38:49,630 --> 00:38:52,590 And so when I built these the the 983 00:38:52,590 --> 00:38:53,730 security configuration 984 00:38:54,190 --> 00:38:55,809 behind these OOB networks, 985 00:38:57,150 --> 00:38:57,650 I 986 00:38:58,110 --> 00:39:01,809 combined the Linux Netfilter Framework Packet Flow 987 00:39:02,235 --> 00:39:02,735 knowledge 988 00:39:03,515 --> 00:39:05,055 with the ideology 989 00:39:05,515 --> 00:39:08,735 of defense in-depth, you know, the military strategy 990 00:39:08,795 --> 00:39:11,695 for defense in-depth. Right? So I I combine 991 00:39:11,755 --> 00:39:13,135 these two approaches 992 00:39:13,675 --> 00:39:16,175 into designing the net sec aspect 993 00:39:16,920 --> 00:39:19,099 of the OOB network. Now, 994 00:39:19,480 --> 00:39:20,460 of course, realistically 995 00:39:20,760 --> 00:39:21,900 realistically speaking, 996 00:39:22,280 --> 00:39:24,519 most most of the businesses out there, they 997 00:39:24,519 --> 00:39:26,300 they don't have the required 998 00:39:27,079 --> 00:39:28,539 in house expertise. 999 00:39:29,000 --> 00:39:30,679 And what I mean by that is normally 1000 00:39:30,679 --> 00:39:31,659 a software engineer, 1001 00:39:32,434 --> 00:39:35,315 to build a CICD pipeline to automate these 1002 00:39:35,315 --> 00:39:37,954 OOB networks and the production network as part 1003 00:39:37,954 --> 00:39:38,534 of it. 1004 00:39:39,074 --> 00:39:40,755 So in a lot of these cases, these 1005 00:39:40,755 --> 00:39:43,394 these these networks are are still manual driven. 1006 00:39:43,394 --> 00:39:44,835 I mean, as much as some of many 1007 00:39:44,835 --> 00:39:46,195 of us would hate to admit it, but 1008 00:39:46,195 --> 00:39:48,239 a lot of networks are still manually managed. 1009 00:39:48,239 --> 00:39:50,260 Automation is not the norm. It's the exception, 1010 00:39:50,480 --> 00:39:50,980 unfortunately. 1011 00:39:51,280 --> 00:39:52,500 It's still very expensive, 1012 00:39:53,280 --> 00:39:55,280 to hire a expert who can build a 1013 00:39:55,280 --> 00:39:56,260 CICD pipeline, 1014 00:39:56,960 --> 00:39:58,340 for your automation. So, 1015 00:39:59,119 --> 00:40:00,659 when I build these security 1016 00:40:00,960 --> 00:40:02,179 configuration parameters, 1017 00:40:02,894 --> 00:40:06,035 I make good use of the prerouting chain 1018 00:40:06,494 --> 00:40:09,295 on on these Linux based device network devices. 1019 00:40:09,295 --> 00:40:10,974 So in the out of band network, I'm 1020 00:40:11,135 --> 00:40:13,135 I I some might say I'm biased, but 1021 00:40:13,135 --> 00:40:13,635 whatever. 1022 00:40:14,015 --> 00:40:17,449 I'm biased to use Linux centric network devices 1023 00:40:17,590 --> 00:40:20,250 or operating systems in the OOB network. Why? 1024 00:40:20,630 --> 00:40:22,489 Because I can take a lot of advantage 1025 00:40:22,550 --> 00:40:23,289 with the prerouting 1026 00:40:23,989 --> 00:40:26,550 chain. So if if if you recall in 1027 00:40:26,550 --> 00:40:29,190 the in in the last twenty plus years, 1028 00:40:29,190 --> 00:40:30,250 there have been 1029 00:40:30,550 --> 00:40:31,449 very advanced 1030 00:40:32,324 --> 00:40:32,984 low level, 1031 00:40:33,605 --> 00:40:36,425 network attacks that takes advantage of how the 1032 00:40:36,724 --> 00:40:39,764 pack, the packet processing engine works on these 1033 00:40:39,764 --> 00:40:42,824 various operating systems. Right? Like, they could construct 1034 00:40:42,965 --> 00:40:45,224 a malformed packet with a certain 1035 00:40:45,609 --> 00:40:47,690 fragment in a certain way with a certain 1036 00:40:47,690 --> 00:40:49,630 byte size and do some real damage. 1037 00:40:49,929 --> 00:40:52,570 But if you drop unauthorized packets in the 1038 00:40:52,570 --> 00:40:54,489 pre routing chain, it doesn't even get very 1039 00:40:54,489 --> 00:40:56,329 far. Right? It gets dropped I mean, it 1040 00:40:56,329 --> 00:40:58,809 does pass the socket buffer, SK buffer of 1041 00:40:58,809 --> 00:40:59,869 the Linux framework, 1042 00:41:00,515 --> 00:41:02,355 but it doesn't get very far because you're 1043 00:41:02,355 --> 00:41:04,275 dropping it in the pre routing chain, so 1044 00:41:04,275 --> 00:41:06,454 it doesn't even enter the connection tracking table 1045 00:41:06,515 --> 00:41:08,755 whatsoever. It doesn't go there. So if there's 1046 00:41:08,755 --> 00:41:10,594 any kind of attack that is trying to 1047 00:41:10,594 --> 00:41:11,335 do something 1048 00:41:11,715 --> 00:41:13,555 post the pre routing chain, it's not gonna 1049 00:41:13,555 --> 00:41:15,299 happen because my prerouting, 1050 00:41:15,839 --> 00:41:16,339 prerouting 1051 00:41:16,719 --> 00:41:18,019 filter rules or ACLs 1052 00:41:18,719 --> 00:41:21,039 are dropping those package the moment they they 1053 00:41:21,039 --> 00:41:22,559 hit the NIC or the moment they they 1054 00:41:22,559 --> 00:41:23,780 process by the system. 1055 00:41:25,119 --> 00:41:26,880 So I, of course, like I had mentioned 1056 00:41:26,880 --> 00:41:29,505 earlier, segmentation is important and decoupling 1057 00:41:32,704 --> 00:41:34,944 topology in the outbound network that is kind 1058 00:41:34,944 --> 00:41:36,164 of similar to 1059 00:41:36,864 --> 00:41:39,105 a collapsed core topology, sort of similar to 1060 00:41:39,105 --> 00:41:40,244 that. So you have the 1061 00:41:40,545 --> 00:41:43,380 OOB edge router, which has your upstream connectivity, 1062 00:41:43,380 --> 00:41:45,699 whether that's a dedicated transit, that that's your 1063 00:41:45,699 --> 00:41:48,260 Starling, LTE, whatever you want. That's where the 1064 00:41:48,260 --> 00:41:51,139 OOB Edge is. And below that is my 1065 00:41:51,139 --> 00:41:53,380 OOB layer three distribution router as I call 1066 00:41:53,380 --> 00:41:54,980 it. Some people might call it something else. 1067 00:41:54,980 --> 00:41:56,440 The name is not really important. 1068 00:41:56,844 --> 00:41:58,765 And and on the on the layer view 1069 00:41:58,765 --> 00:42:00,465 distributions where I have the stateful, 1070 00:42:01,565 --> 00:42:04,625 firewall rules to protect my OOB client devices. 1071 00:42:04,684 --> 00:42:05,664 So that includes, 1072 00:42:07,244 --> 00:42:09,905 systems, physical server boxes maybe, 1073 00:42:10,920 --> 00:42:13,400 your other network devices, or and even your 1074 00:42:13,400 --> 00:42:16,460 PDUs. So your your your power your electrical 1075 00:42:16,519 --> 00:42:19,260 components that have Ethernet management, 1076 00:42:19,880 --> 00:42:23,079 those are also protected with the OOB, and 1077 00:42:23,079 --> 00:42:25,099 you can access those through the OOB. 1078 00:42:26,154 --> 00:42:27,914 So it could it's not just the network. 1079 00:42:27,914 --> 00:42:29,434 It's more than the network. You can have 1080 00:42:29,434 --> 00:42:31,914 IoT devices, you know, like door controls, that 1081 00:42:31,914 --> 00:42:34,554 kind of stuff, building control, access control. All 1082 00:42:34,554 --> 00:42:36,315 all that kind of stuff should be tied 1083 00:42:36,315 --> 00:42:38,394 down to the OOB network, not to the 1084 00:42:38,394 --> 00:42:39,934 production revenue network. 1085 00:42:40,500 --> 00:42:43,219 And, yes, security wise, they are protected by 1086 00:42:43,219 --> 00:42:45,239 a stateful firewall on the late fee distribution. 1087 00:42:45,780 --> 00:42:48,039 And the OOB devices themselves, 1088 00:42:48,500 --> 00:42:50,840 since they have rules in the prerouting chain, 1089 00:42:50,900 --> 00:42:54,164 they are themselves protected on ingress from any 1090 00:42:54,164 --> 00:42:56,485 kind of potential malformed packet that may be 1091 00:42:56,485 --> 00:42:57,864 trying to take advantage of 1092 00:42:58,164 --> 00:43:00,585 undisclosed, undiscovered security vulnerabilities 1093 00:43:02,005 --> 00:43:03,144 in the Linux kernel. 1094 00:43:03,925 --> 00:43:05,844 So that's how I tackle it. Of course, 1095 00:43:05,844 --> 00:43:07,445 there's a lot more involved once you go 1096 00:43:07,445 --> 00:43:09,389 to layer seven. Right? You have zero trust 1097 00:43:09,389 --> 00:43:11,150 and all these things, and it gets really 1098 00:43:11,150 --> 00:43:13,730 complicated. But but the principle is that, 1099 00:43:14,109 --> 00:43:16,589 I I really am biased with a Linux 1100 00:43:16,589 --> 00:43:17,489 centric approach, 1101 00:43:18,349 --> 00:43:20,269 but maybe there could be better approach that 1102 00:43:20,269 --> 00:43:21,630 I'm not aware of. But so far, for 1103 00:43:21,630 --> 00:43:23,809 me, Linux centric approach for OOBE 1104 00:43:24,574 --> 00:43:26,514 has paid off. It has worked pretty well. 1105 00:43:26,894 --> 00:43:28,894 It's actually probably less expensive too because you 1106 00:43:28,894 --> 00:43:31,234 can actually run Linux on fairly small boxes. 1107 00:43:31,454 --> 00:43:33,454 I mean, even down to some larger Raspberry 1108 00:43:33,454 --> 00:43:35,534 Pis, if you were that crazy about doing 1109 00:43:35,534 --> 00:43:38,590 it, that would give you a very inexpensive 1110 00:43:38,890 --> 00:43:40,570 way, and then your main con your main 1111 00:43:40,570 --> 00:43:41,789 problem becomes connectivity 1112 00:43:42,489 --> 00:43:43,630 into the back end 1113 00:43:44,010 --> 00:43:44,909 rather than, 1114 00:43:45,610 --> 00:43:47,789 you know, main managing and maintaining 1115 00:43:48,570 --> 00:43:51,449 large expensive boxes with spinning hard drives and 1116 00:43:51,449 --> 00:43:53,025 stuff like that. Yeah. 1117 00:43:54,065 --> 00:43:55,184 So I don't know. I feel like we've 1118 00:43:55,184 --> 00:43:56,704 kind of beat this to death. Is there 1119 00:43:56,704 --> 00:43:58,864 anything else you wanna talk about, Daryl, before 1120 00:43:58,864 --> 00:43:59,684 we wrap up? 1121 00:44:00,625 --> 00:44:03,664 No. I think we covered the the main 1122 00:44:03,664 --> 00:44:06,484 principles, I think, of the, out of band, 1123 00:44:06,969 --> 00:44:07,710 you know, 1124 00:44:08,010 --> 00:44:10,170 business wise and technical wise, I think we 1125 00:44:10,170 --> 00:44:12,890 covered most of the concepts. So, you know, 1126 00:44:12,890 --> 00:44:14,969 may maybe Tom has some something to say. 1127 00:44:14,969 --> 00:44:15,469 Yeah? 1128 00:44:16,170 --> 00:44:18,349 Tom always has something to say. He says 1129 00:44:18,650 --> 00:44:19,150 LinkedIn. 1130 00:44:20,054 --> 00:44:20,554 That's 1131 00:44:21,014 --> 00:44:23,094 right. But I'm saving I'm saving that for 1132 00:44:23,094 --> 00:44:23,994 the right moment. 1133 00:44:28,614 --> 00:44:29,355 Oh, boy. 1134 00:44:29,974 --> 00:44:32,534 Alright. Well, Daryl, where can people find you? 1135 00:44:32,534 --> 00:44:34,679 Do you blog or anything else? I I 1136 00:44:34,679 --> 00:44:36,199 saw your blog post on this, so you 1137 00:44:36,199 --> 00:44:37,739 must have an outlet. So 1138 00:44:38,039 --> 00:44:39,480 do you blog on a regular basis, 1139 00:44:40,280 --> 00:44:42,859 or do you Twitter? Yes. Whatever? 1140 00:44:44,519 --> 00:44:46,440 I do publish blog posts now and then. 1141 00:44:46,440 --> 00:44:47,819 I do have my own website. 1142 00:44:48,585 --> 00:44:50,985 It goes by my name, deaddelsware.com. 1143 00:44:50,985 --> 00:44:52,344 You can find me there. And on my 1144 00:44:52,344 --> 00:44:54,344 website, you can find links to other social 1145 00:44:54,344 --> 00:44:58,204 platforms, LinkedIn, Twitter, etcetera, etcetera. Yeah. Okay. Awesome. 1146 00:44:58,824 --> 00:45:00,719 And go ahead, Tom. Just say it. 1147 00:45:01,199 --> 00:45:01,699 LinkedIn. 1148 00:45:03,920 --> 00:45:04,739 I've been practicing. 1149 00:45:08,239 --> 00:45:10,320 I I don't think you're fast enough. K. 1150 00:45:10,320 --> 00:45:11,380 I'll work on it. 1151 00:45:13,440 --> 00:45:15,519 Alright. I'm Russ White. You can always find 1152 00:45:15,519 --> 00:45:17,755 me here at the hedge, rule eleven dot 1153 00:45:18,055 --> 00:45:20,954 tech. I log into x occasionally 1154 00:45:21,255 --> 00:45:23,735 and LinkedIn occasionally, so you might find me 1155 00:45:23,735 --> 00:45:24,235 there, 1156 00:45:24,695 --> 00:45:26,055 you know, if you wanna DM me or 1157 00:45:26,055 --> 00:45:27,894 PM me in one of those places. If 1158 00:45:27,894 --> 00:45:29,275 you have any ideas 1159 00:45:29,699 --> 00:45:31,380 for things you'd like to hear or talk 1160 00:45:31,380 --> 00:45:33,159 about hear us talk about on the hedge, 1161 00:45:33,460 --> 00:45:34,440 please let me know. 1162 00:45:35,139 --> 00:45:36,980 And, Daryl, thanks for coming. By the way, 1163 00:45:36,980 --> 00:45:38,739 you know, anytime you have another topic or 1164 00:45:38,739 --> 00:45:40,420 something you wanna talk about, please just let 1165 00:45:40,420 --> 00:45:42,659 us know. We're pretty open to new ideas 1166 00:45:42,659 --> 00:45:44,444 and thoughts. And I don't know if you 1167 00:45:44,444 --> 00:45:46,744 know this, but podcasts like eat material. 1168 00:45:49,364 --> 00:45:51,304 That's it's it's always, 1169 00:45:51,764 --> 00:45:53,304 we can always use more topics. 1170 00:45:53,684 --> 00:45:54,664 Yeah. They do. 1171 00:45:55,525 --> 00:45:56,025 So 1172 00:45:56,405 --> 00:45:56,905 alright. 1173 00:45:57,680 --> 00:45:59,119 Thank we'd like to thank you for listening. 1174 00:45:59,119 --> 00:46:00,400 We know that we live in an attention 1175 00:46:00,400 --> 00:46:02,960 driven economy, so your attention's probably the most 1176 00:46:02,960 --> 00:46:06,000 economically valuable thing out there right now. So 1177 00:46:06,000 --> 00:46:07,860 thanks for spending the time with us, 1178 00:46:08,160 --> 00:46:08,660 and 1179 00:46:09,200 --> 00:46:10,660 we will catch you next time. 1180 00:46:18,184 --> 00:46:18,684 Time.