1 00:00:00,080 --> 00:00:02,020 Join us at the hedge for a conversation 2 00:00:02,159 --> 00:00:03,060 about engineering, 3 00:00:03,439 --> 00:00:07,040 technology, and business. This week, Russ White, Tom 4 00:00:07,040 --> 00:00:09,939 Ammon, and Derek Winkworth dig into automation. 5 00:00:21,285 --> 00:00:22,744 So, Tom, how are you? 6 00:00:23,445 --> 00:00:25,765 Doing good. Ready for the weekend. Ready for 7 00:00:25,765 --> 00:00:27,625 the weekend. Derek, how are you? 8 00:00:29,170 --> 00:00:31,490 I'm good. I'm I'm also ready for the 9 00:00:31,490 --> 00:00:33,649 weekend. I, although I probably won't do a 10 00:00:33,649 --> 00:00:35,570 lot outside because it's supposed to be crazy 11 00:00:35,570 --> 00:00:36,310 hot tomorrow 12 00:00:37,490 --> 00:00:39,570 Yeah, I know I know it is that 13 00:00:39,570 --> 00:00:41,545 time of year and that heat wave has 14 00:00:41,545 --> 00:00:43,465 been coming through and really it's kind of 15 00:00:43,465 --> 00:00:45,864 crazy how hot it's been Yeah, I just 16 00:00:45,864 --> 00:00:47,704 got off. I just did a three hour 17 00:00:47,704 --> 00:00:50,664 webinar on data center design for Pearson So 18 00:00:50,664 --> 00:00:52,585 I just got off that recording. So if 19 00:00:52,585 --> 00:00:55,405 my voice sounds a little off, that's because 20 00:00:55,820 --> 00:00:57,119 it's a little off. 21 00:00:58,219 --> 00:01:00,299 You've been talking for three hours. I've been 22 00:01:00,299 --> 00:01:01,359 talking for three. 23 00:01:01,820 --> 00:01:03,200 It's a little bit crazy. 24 00:01:04,060 --> 00:01:06,000 So, Derek, let's back up. You 25 00:01:06,700 --> 00:01:08,620 you talk about your days as a network 26 00:01:08,620 --> 00:01:10,859 engineer in the for in the far distant 27 00:01:10,859 --> 00:01:11,359 past 28 00:01:11,715 --> 00:01:13,415 before your current life 29 00:01:15,234 --> 00:01:16,754 Yeah, you're one of the smart ones you 30 00:01:16,754 --> 00:01:18,674 got out of network engineering before it drove 31 00:01:18,674 --> 00:01:19,334 you crazy 32 00:01:24,819 --> 00:01:26,200 What did I do to myself? 33 00:01:27,619 --> 00:01:29,560 Oh, man. Yeah. It's true. 34 00:01:29,859 --> 00:01:32,340 It it is a very stressful job, and 35 00:01:32,340 --> 00:01:34,899 it's and it's insane. Like, the being 36 00:01:35,700 --> 00:01:37,239 choosing to be a network engineer 37 00:01:37,775 --> 00:01:40,334 is, it's it kind of is an insane 38 00:01:40,334 --> 00:01:41,635 thing to choose to be, 39 00:01:42,734 --> 00:01:43,474 for sure. 40 00:01:44,094 --> 00:01:45,694 So talk to me about automation. You start 41 00:01:45,775 --> 00:01:47,454 we started talking about automation a little bit 42 00:01:47,454 --> 00:01:48,895 in the life of a network engineer, so 43 00:01:48,895 --> 00:01:50,355 let's back up your your experience. 44 00:01:50,829 --> 00:01:53,229 Because I know you mostly do automation today. 45 00:01:53,229 --> 00:01:55,329 Right? That's mostly where you work is around 46 00:01:55,469 --> 00:01:57,009 automation and cloud stuff. 47 00:01:57,549 --> 00:01:58,049 Yep. 48 00:01:58,909 --> 00:02:00,530 Yeah. Yeah. So I, 49 00:02:01,709 --> 00:02:03,644 that that's what I do, I guess. I 50 00:02:04,284 --> 00:02:04,784 I, 51 00:02:05,644 --> 00:02:07,084 the reason why I sort of left the 52 00:02:07,084 --> 00:02:08,764 customer side and went to the vendor side 53 00:02:08,764 --> 00:02:11,164 was is in part was entirely because of 54 00:02:11,164 --> 00:02:13,164 automation. That was right around the time that 55 00:02:13,164 --> 00:02:15,004 SDN was in its, you know, at the 56 00:02:15,004 --> 00:02:17,250 peak of its hype cycle that I decided 57 00:02:17,250 --> 00:02:17,650 to, 58 00:02:18,289 --> 00:02:20,209 decided to come to the vendor side, the 59 00:02:20,209 --> 00:02:20,949 dark side. 60 00:02:21,569 --> 00:02:22,069 And, 61 00:02:23,329 --> 00:02:25,969 part of my motivation was whenever I talk 62 00:02:25,969 --> 00:02:28,289 to people about network automation on the vendor 63 00:02:28,289 --> 00:02:30,314 side or even on the customer side, then 64 00:02:30,314 --> 00:02:31,834 one of the things that kinda blew my 65 00:02:31,834 --> 00:02:32,334 mind 66 00:02:32,634 --> 00:02:34,875 is they wanted to they wanted an automated 67 00:02:34,875 --> 00:02:35,375 network. 68 00:02:36,074 --> 00:02:37,935 And when you ask them what that meant, 69 00:02:39,114 --> 00:02:41,055 you know, they I love that. An automated 70 00:02:41,194 --> 00:02:43,034 network. Yeah. I just I want an automated 71 00:02:43,034 --> 00:02:44,289 network. I don't know. Yeah. What does that 72 00:02:44,289 --> 00:02:46,629 even mean? Right? So you would ask them, 73 00:02:46,849 --> 00:02:48,849 and they would say, well, if this thing 74 00:02:48,849 --> 00:02:50,689 happens in the network, then I want some 75 00:02:50,689 --> 00:02:53,569 system that'll automatically go and and do some 76 00:02:53,569 --> 00:02:54,050 kind of, 77 00:02:54,689 --> 00:02:57,030 configuration change or some kind of action that'll 78 00:02:57,090 --> 00:02:58,629 alter the behavior of the network. 79 00:02:59,645 --> 00:03:02,125 And I thought that was insane, right? Because 80 00:03:02,125 --> 00:03:04,625 you have this networks are already very complicated, 81 00:03:04,685 --> 00:03:06,784 and they're built on fragile components. 82 00:03:07,645 --> 00:03:08,145 And, 83 00:03:08,525 --> 00:03:11,004 you know, they're they're already very dynamic by 84 00:03:11,004 --> 00:03:13,004 their very nature, right? They I mean, we 85 00:03:13,004 --> 00:03:16,099 have dynamic protocols and, you know, every everything 86 00:03:16,099 --> 00:03:18,500 is dynamic. Right? The way every everything in 87 00:03:18,500 --> 00:03:21,240 the network works. When you have bonded interfaces, 88 00:03:21,379 --> 00:03:22,840 right, they they 89 00:03:23,140 --> 00:03:24,819 go down and come back up and the 90 00:03:24,819 --> 00:03:25,640 system automatically, 91 00:03:26,980 --> 00:03:29,140 adjusts, you know, how it forwards according to 92 00:03:29,140 --> 00:03:30,599 those the state of the interface. 93 00:03:30,935 --> 00:03:33,034 Like, everything is is already automatic. 94 00:03:33,575 --> 00:03:36,215 So when you wanna add more dynamic behavior 95 00:03:36,215 --> 00:03:38,314 on top of a system that's already dynamic 96 00:03:38,855 --> 00:03:41,335 and which already, if you if you stop 97 00:03:41,335 --> 00:03:43,655 to think about it, is is already barely 98 00:03:43,655 --> 00:03:44,155 functions. 99 00:03:44,540 --> 00:03:46,060 There is no such thing as a network 100 00:03:46,060 --> 00:03:48,699 that you can ignore because it it'll always 101 00:03:48,699 --> 00:03:49,840 fail in some way. 102 00:03:50,219 --> 00:03:52,379 And now you wanna add more dynamic behavior 103 00:03:52,379 --> 00:03:54,080 on top of that. And to me, 104 00:03:55,740 --> 00:03:58,240 that that sounds crazy. Right? And I 105 00:03:58,645 --> 00:04:00,405 and I, and I, what we ought to 106 00:04:00,405 --> 00:04:02,324 be automating in my mind is what I 107 00:04:02,324 --> 00:04:04,164 always thought was we ought to be automating 108 00:04:04,164 --> 00:04:06,185 what network engineers actually do. 109 00:04:06,645 --> 00:04:07,145 And, 110 00:04:07,685 --> 00:04:08,344 and therein 111 00:04:08,965 --> 00:04:11,039 lies the gist, you know, if, or the 112 00:04:11,120 --> 00:04:12,799 devil's in the details, as they say. If 113 00:04:12,799 --> 00:04:14,739 you ask a network engineer what they do, 114 00:04:14,799 --> 00:04:16,740 they'll say things like, well, I configure BGP, 115 00:04:16,800 --> 00:04:18,819 and or, you know, I build 116 00:04:20,639 --> 00:04:23,839 data center networks, or I configure VLANs, or 117 00:04:23,839 --> 00:04:26,485 whatever they say. And if you ask, like 118 00:04:26,485 --> 00:04:28,964 a typical product manager at a network vendor 119 00:04:28,964 --> 00:04:30,425 what network engineers do, 120 00:04:31,125 --> 00:04:34,185 they'll reiterate that. And there's so there's this, 121 00:04:34,404 --> 00:04:35,064 you know, 122 00:04:36,004 --> 00:04:38,324 unrealistic perception about what we ought to be 123 00:04:38,324 --> 00:04:40,105 automating because people don't actually 124 00:04:41,360 --> 00:04:42,960 they they actually have no clue what network 125 00:04:42,960 --> 00:04:45,379 engineers actually do, including network engineers themselves. 126 00:04:46,160 --> 00:04:48,180 I was gonna say, don't yeah. Yeah. 127 00:04:51,439 --> 00:04:53,120 That's right. Because we don't know what we 128 00:04:53,120 --> 00:04:54,100 do for a living. 129 00:04:56,855 --> 00:04:58,775 Yeah. It it's true. And if you you 130 00:04:58,775 --> 00:05:01,175 know what is crazy about that statement? I 131 00:05:01,175 --> 00:05:02,694 mean, just to go a little bit outside 132 00:05:02,694 --> 00:05:05,335 the automation realm is that I've never been 133 00:05:05,335 --> 00:05:07,735 a network engineer anywhere where I only did 134 00:05:07,735 --> 00:05:10,694 network engineering. Right? I I always had to 135 00:05:10,694 --> 00:05:11,160 do 136 00:05:11,560 --> 00:05:12,220 some other, 137 00:05:12,680 --> 00:05:13,660 residual set 138 00:05:14,040 --> 00:05:16,699 of things, whether that was actually troubleshooting applications, 139 00:05:17,959 --> 00:05:19,339 you know, maintaining, 140 00:05:19,639 --> 00:05:21,720 like, a server somewhere that no one else 141 00:05:21,720 --> 00:05:24,145 wants to maintain. Right? The UNIX guys will 142 00:05:24,145 --> 00:05:25,824 stand the server up, and and it has 143 00:05:25,824 --> 00:05:28,944 some function like DNS or or, network a 144 00:05:28,944 --> 00:05:31,345 network management system, and they're not gonna no 145 00:05:31,345 --> 00:05:32,884 one wants to manage that. So, 146 00:05:33,904 --> 00:05:35,824 you always end up you always have ancillary 147 00:05:35,824 --> 00:05:37,665 duties as a as a network engineer. So 148 00:05:37,665 --> 00:05:39,480 that that muddies the water even more. 149 00:05:40,839 --> 00:05:43,720 Oh, definitely. Yeah. Definitely. Yeah. That that's that's 150 00:05:43,720 --> 00:05:44,939 an interesting observation. 151 00:05:45,240 --> 00:05:46,600 And I think a lot of people don't 152 00:05:46,600 --> 00:05:48,439 think about that. Right? They think, well, you 153 00:05:48,439 --> 00:05:50,839 just do network engineering. But in reality, you 154 00:05:50,839 --> 00:05:53,080 are maintaining all these odd servers and all 155 00:05:53,080 --> 00:05:55,185 this stuff that people have put out there, 156 00:05:55,185 --> 00:05:56,564 and you're just like, I 157 00:05:57,104 --> 00:05:58,384 I don't know. How did I end up 158 00:05:58,384 --> 00:06:00,144 being a sysadmin? I don't know, but I 159 00:06:00,144 --> 00:06:00,644 did. 160 00:06:02,305 --> 00:06:04,224 You you know what I've always wondered? And 161 00:06:04,224 --> 00:06:06,144 with your exposure to customers, Derek, maybe you 162 00:06:06,144 --> 00:06:07,889 have some insights on this. What do you 163 00:06:07,889 --> 00:06:10,770 think what percentage of an engineer network engineer's 164 00:06:10,770 --> 00:06:12,790 workload do they actually really enjoy? 165 00:06:13,090 --> 00:06:15,270 Would you say it's 50% for most engineers? 166 00:06:15,330 --> 00:06:17,830 10? 20? Do they do they enjoy? 167 00:06:19,774 --> 00:06:21,375 Wow. That's a well, I'm sure that's a 168 00:06:21,375 --> 00:06:23,535 very subjective thing, but Yeah. Sure. I, 169 00:06:24,735 --> 00:06:26,895 I would say it's very small. It's very 170 00:06:27,055 --> 00:06:28,175 I'll give you an I'll so you know 171 00:06:28,175 --> 00:06:29,855 what? I'll start off with an analogy, and 172 00:06:29,855 --> 00:06:31,795 then I'll explain explain what I mean. 173 00:06:33,055 --> 00:06:34,675 One of the one of the, 174 00:06:35,399 --> 00:06:38,039 like, moments of joy that I, I was 175 00:06:38,039 --> 00:06:40,759 just overcome with joy and like this immense 176 00:06:40,759 --> 00:06:41,259 satisfaction 177 00:06:41,800 --> 00:06:44,699 in the last few, probably last five years, 178 00:06:44,919 --> 00:06:46,919 was when, I had to build a fence 179 00:06:46,919 --> 00:06:48,774 in my yard because we wanted to install 180 00:06:48,774 --> 00:06:50,794 a pool in our yard. And 181 00:06:51,095 --> 00:06:53,334 the, the town said, well, you gotta have 182 00:06:53,334 --> 00:06:55,274 a fence because it's an attractive nuisance. 183 00:06:55,654 --> 00:06:57,414 And kids, you know, a kid might climb 184 00:06:57,414 --> 00:06:58,875 in the ladder and go in the pool. 185 00:06:59,095 --> 00:06:59,574 Mhmm. So, 186 00:07:00,829 --> 00:07:03,310 we decided to build the the fence ourselves. 187 00:07:03,310 --> 00:07:04,910 I I bought an earth logger. We bought 188 00:07:04,910 --> 00:07:06,349 all the stuff. I had a friend who 189 00:07:06,349 --> 00:07:08,110 had some experience building a fence. He helped 190 00:07:08,110 --> 00:07:09,870 me. And at the end of it, like, 191 00:07:09,870 --> 00:07:11,889 when we got done with that project, like, 192 00:07:11,949 --> 00:07:13,089 there was the fence. 193 00:07:13,644 --> 00:07:15,724 Right? It was like, oh my gosh, like 194 00:07:15,724 --> 00:07:16,784 I had a project 195 00:07:17,724 --> 00:07:20,044 that actually finished. And I can look at 196 00:07:20,044 --> 00:07:22,204 it and say that thing is I made 197 00:07:22,204 --> 00:07:24,064 that thing, right? And it has a purpose, 198 00:07:24,365 --> 00:07:26,819 has a function. And it's still standing doing 199 00:07:26,819 --> 00:07:28,019 the same thing it was meant to do 200 00:07:28,019 --> 00:07:29,240 at the time I built it. 201 00:07:29,699 --> 00:07:31,319 That is a very rare 202 00:07:32,100 --> 00:07:34,899 occasion in the realm of network engineering, unless 203 00:07:34,899 --> 00:07:35,959 maybe you're a contractor 204 00:07:36,419 --> 00:07:38,759 and you can say, you know, my engagement 205 00:07:38,899 --> 00:07:40,735 is done. But when you work for a 206 00:07:40,735 --> 00:07:41,235 company, 207 00:07:42,175 --> 00:07:44,035 it's it's a pretty rare thing 208 00:07:44,495 --> 00:07:46,115 to to feel like 209 00:07:46,495 --> 00:07:46,995 you 210 00:07:47,535 --> 00:07:50,495 you built that or you, you know, you 211 00:07:50,495 --> 00:07:51,615 did something, right? 212 00:07:52,095 --> 00:07:54,334 Other than getting yelled at and and putting 213 00:07:54,334 --> 00:07:55,560 out fires all the time. 214 00:07:55,959 --> 00:07:58,279 And so I that was I like that 215 00:07:58,279 --> 00:08:01,100 feeling of having having built something. And so, 216 00:08:02,040 --> 00:08:03,959 it you know, if if that when you 217 00:08:03,959 --> 00:08:06,120 get when you're when you go up the 218 00:08:06,120 --> 00:08:07,560 chain as a network engineer and you get 219 00:08:07,560 --> 00:08:08,860 into design and architecture, 220 00:08:09,639 --> 00:08:09,879 those, 221 00:08:11,144 --> 00:08:14,444 that experience becomes, you know, once every 222 00:08:15,144 --> 00:08:17,704 few months, twice a year sometimes, depending on 223 00:08:17,704 --> 00:08:18,444 your projects. 224 00:08:18,824 --> 00:08:21,225 And that's and and even then it's like, 225 00:08:21,625 --> 00:08:23,785 you don't get the same satisfaction because it's 226 00:08:23,785 --> 00:08:24,524 pretty rare 227 00:08:24,879 --> 00:08:27,120 that by the time the project is getting 228 00:08:27,120 --> 00:08:29,519 to implementation that you're actually building the thing 229 00:08:29,519 --> 00:08:31,360 you should have built. Because it's been, you 230 00:08:31,360 --> 00:08:33,600 know, everyone's had an opinion that's been pared 231 00:08:33,600 --> 00:08:35,840 down. You're not you're not actually building it 232 00:08:35,840 --> 00:08:37,519 the right way because they wanted to cut 233 00:08:37,519 --> 00:08:40,845 costs. Or there's other restrictions like, you know, 234 00:08:40,845 --> 00:08:42,925 oh, we're gonna have two routers. Well, actually, 235 00:08:42,925 --> 00:08:45,404 we can't get two routers into this building 236 00:08:45,404 --> 00:08:47,345 because there's physically not enough space, 237 00:08:47,965 --> 00:08:50,205 or not enough power. Right? There's always something 238 00:08:50,205 --> 00:08:52,230 that get and then so you build it 239 00:08:52,230 --> 00:08:54,230 and you're like, oh, you just look at 240 00:08:54,230 --> 00:08:55,830 it and you know, this is going this 241 00:08:55,830 --> 00:08:58,149 is I'm gonna have a relationship with this 242 00:08:58,149 --> 00:09:00,230 installation now for the rest of my career 243 00:09:00,230 --> 00:09:03,509 at this at this point. It's gonna fail 244 00:09:03,509 --> 00:09:04,730 all the time. Right? 245 00:09:05,029 --> 00:09:07,445 So it's it's pretty rare to get that 246 00:09:07,664 --> 00:09:09,745 that a sense of accomplishment or enjoyment, I 247 00:09:09,745 --> 00:09:10,245 think. 248 00:09:11,345 --> 00:09:13,125 But I do enjoy I did enjoy troubleshooting, 249 00:09:14,065 --> 00:09:15,504 quite a bit because I was I was 250 00:09:15,504 --> 00:09:16,644 good at it. And, 251 00:09:17,024 --> 00:09:18,945 I could troubleshoot all kinds of odd things 252 00:09:18,945 --> 00:09:21,040 like voice systems and, you know, things that 253 00:09:21,040 --> 00:09:22,899 I don't even necessarily have training in, 254 00:09:23,840 --> 00:09:24,740 because I, 255 00:09:25,120 --> 00:09:26,639 I don't it's just something I like doing. 256 00:09:26,639 --> 00:09:28,160 Right? I like digging rabbit holes until I 257 00:09:28,160 --> 00:09:30,160 figure out what's going on. Just and the 258 00:09:30,160 --> 00:09:31,440 reason I guess the reason I asked that 259 00:09:31,440 --> 00:09:32,799 is I was reading through some of your 260 00:09:32,799 --> 00:09:33,600 blog posts and, 261 00:09:34,264 --> 00:09:35,705 talking about how we do a lot of, 262 00:09:35,705 --> 00:09:38,664 you know, mental modeling as network engineers. You 263 00:09:38,664 --> 00:09:40,264 know, we build models to try and figure 264 00:09:40,264 --> 00:09:42,284 out what happened or what's going to happen. 265 00:09:43,465 --> 00:09:45,065 A lot of the network engineers that I 266 00:09:45,065 --> 00:09:46,200 talked to, it seems like 267 00:09:47,000 --> 00:09:49,080 they like automation because they get to build 268 00:09:49,080 --> 00:09:51,399 something again. Instead of in my experience, it's 269 00:09:51,399 --> 00:09:53,320 exactly what you're talking about. You don't actually 270 00:09:53,320 --> 00:09:55,160 get a build very much when it comes 271 00:09:55,160 --> 00:09:57,240 down to it, you know, especially the more 272 00:09:57,240 --> 00:09:58,215 senior you get. 273 00:09:59,175 --> 00:10:01,415 But it seems like automation gives us a 274 00:10:01,415 --> 00:10:03,735 chance to to build something and complete something. 275 00:10:03,735 --> 00:10:05,815 And in most companies, automation, at least my 276 00:10:05,815 --> 00:10:07,975 experience, is never done. You're always going to 277 00:10:07,975 --> 00:10:10,075 be building that thing and tweaking it and 278 00:10:10,295 --> 00:10:11,700 all that kind of stuff. So, 279 00:10:12,259 --> 00:10:14,419 do you does that resonate with other people? 280 00:10:14,419 --> 00:10:15,940 I mean, is that why engineers like to 281 00:10:15,940 --> 00:10:17,460 do automation, do you think? Or are there 282 00:10:17,460 --> 00:10:20,580 other other things? Yeah. I think so. That's 283 00:10:20,580 --> 00:10:23,139 true. You get to what I well, what 284 00:10:23,139 --> 00:10:24,964 I like about the current sort of strain 285 00:10:24,964 --> 00:10:27,205 of automation, when when SDN got huge, right, 286 00:10:27,205 --> 00:10:28,024 it was all 287 00:10:28,404 --> 00:10:30,804 like network engineers are the mainframe programmers of 288 00:10:30,804 --> 00:10:32,964 the day. Right? And and if you remember, 289 00:10:33,605 --> 00:10:35,125 hold on. I always forget the name of 290 00:10:35,125 --> 00:10:36,644 the company. The the one that got bought 291 00:10:36,644 --> 00:10:39,570 by VMware that became NSX. This was, Nicero. 292 00:10:39,570 --> 00:10:41,029 Nicero. Yeah. Nicero. 293 00:10:41,649 --> 00:10:43,909 They they actually would tell customers, 294 00:10:45,169 --> 00:10:45,829 you know, 295 00:10:46,529 --> 00:10:48,529 do not invite the network engineer to this 296 00:10:48,529 --> 00:10:50,049 meeting. And they would talk to the server 297 00:10:50,049 --> 00:10:50,549 guys. 298 00:10:51,105 --> 00:10:53,345 And because they because they're they're irrelevant now, 299 00:10:53,345 --> 00:10:54,945 or you don't need them. Right? That's that's 300 00:10:55,184 --> 00:10:56,245 that was their message. 301 00:10:56,705 --> 00:10:58,465 It was it was crazy, and it and 302 00:10:58,465 --> 00:11:00,544 that's still a thing in Silicon Valley. There 303 00:11:00,544 --> 00:11:02,304 are companies that really believe they're going to 304 00:11:02,304 --> 00:11:04,789 obviate the need for network engineers. Oh, I 305 00:11:04,789 --> 00:11:06,629 I know some I know some companies that 306 00:11:06,629 --> 00:11:09,129 refuse to hire network engineers in the future. 307 00:11:09,750 --> 00:11:12,149 You know, they're they're shape reshaping their teams 308 00:11:12,149 --> 00:11:12,809 to be 309 00:11:13,429 --> 00:11:15,589 coders. They only hire coders because they don't 310 00:11:15,589 --> 00:11:17,750 really think they need network engineers any longer, 311 00:11:17,750 --> 00:11:17,866 and they run networks, big ones. Mhmm. That's, 312 00:11:17,866 --> 00:11:18,490 like, that's okay. Mhmm. And 313 00:11:18,845 --> 00:11:19,585 big ones. 314 00:11:20,684 --> 00:11:22,625 That's, like, that's okay. 315 00:11:23,004 --> 00:11:24,924 K. That's a but you know what? GitHub 316 00:11:24,924 --> 00:11:26,845 is one of the examples that people used 317 00:11:26,845 --> 00:11:28,125 to love the town. Like, we don't have 318 00:11:28,125 --> 00:11:30,045 any network engineers for our network, and now 319 00:11:30,045 --> 00:11:31,665 they have them. Now they have several, 320 00:11:33,565 --> 00:11:34,230 because it's 321 00:11:34,629 --> 00:11:36,309 you need that expertise. And what I what 322 00:11:36,309 --> 00:11:38,149 I like about the current crop of automation 323 00:11:38,149 --> 00:11:40,149 is that you've gotten a little bit away 324 00:11:40,149 --> 00:11:42,889 from the, you know, you're irrelevant to, 325 00:11:43,269 --> 00:11:45,269 you know what, we need to start automating 326 00:11:45,269 --> 00:11:47,529 what network engineers actually do. And, 327 00:11:48,149 --> 00:11:50,205 and, of course, we're gonna need people who 328 00:11:50,205 --> 00:11:50,705 understand 329 00:11:51,085 --> 00:11:51,904 network engineering 330 00:11:52,605 --> 00:11:53,504 to do it right. 331 00:11:53,884 --> 00:11:55,264 So it's, I 332 00:11:56,125 --> 00:11:58,684 I think there's I think there's enthusiasm around 333 00:11:58,684 --> 00:12:00,845 that message for sure. So that's interesting. Let's 334 00:12:00,845 --> 00:12:02,699 go back up because you said, you know, 335 00:12:02,699 --> 00:12:05,339 what does a network engineer actually do? And 336 00:12:05,339 --> 00:12:07,019 we don't actually know what we do. So 337 00:12:07,019 --> 00:12:08,699 let's talk about that in automation a little 338 00:12:08,699 --> 00:12:10,299 bit. I mean, how does that how does 339 00:12:10,299 --> 00:12:11,579 that work out? I mean, what do we 340 00:12:11,740 --> 00:12:13,100 what what is it we should be trying 341 00:12:13,100 --> 00:12:14,639 to automate if it's not configuration? 342 00:12:15,899 --> 00:12:16,799 Yeah. So, 343 00:12:17,855 --> 00:12:19,615 that well, the modeling thing I was talking 344 00:12:19,615 --> 00:12:21,615 about earlier. Right? I and one of one 345 00:12:21,615 --> 00:12:23,214 of my I'm you know, we need to 346 00:12:23,214 --> 00:12:25,294 republish these blog posts because everyone listening to 347 00:12:25,294 --> 00:12:25,794 this, 348 00:12:26,575 --> 00:12:27,934 if you're out there, none of there none 349 00:12:27,934 --> 00:12:29,454 of them are on the Internet anymore because 350 00:12:29,454 --> 00:12:31,054 it was I wrote these at a previous 351 00:12:31,054 --> 00:12:33,179 company and that website's gone. But I 352 00:12:34,139 --> 00:12:36,379 I'll republish these. Actually, I'll work on, I'll 353 00:12:36,379 --> 00:12:38,240 rewrite these and republish them. 354 00:12:39,659 --> 00:12:41,339 The, one of the blog posts I wrote 355 00:12:41,339 --> 00:12:43,919 was what a network engineer actually does. 356 00:12:44,299 --> 00:12:44,799 And 357 00:12:45,774 --> 00:12:47,695 I was sort of inspired, I have this, 358 00:12:48,414 --> 00:12:50,414 I have a bunch of books on, like 359 00:12:50,414 --> 00:12:51,634 the psychology of 360 00:12:52,335 --> 00:12:54,335 thinking, like how do and there's one of 361 00:12:54,335 --> 00:12:56,335 those books is almost a whole book, every 362 00:12:56,335 --> 00:12:58,434 chapter they talk about engineers in particular. 363 00:12:58,750 --> 00:13:00,910 These, and these are psychologists, right? Like how 364 00:13:00,910 --> 00:13:01,490 do they, 365 00:13:02,269 --> 00:13:05,149 like conceptually model in their mind the things 366 00:13:05,149 --> 00:13:06,450 that they work on, right? 367 00:13:06,950 --> 00:13:07,450 And, 368 00:13:08,830 --> 00:13:09,889 I, and 369 00:13:10,750 --> 00:13:13,245 from that, I started really thinking about what, 370 00:13:13,325 --> 00:13:16,044 what engineers what network engineers do. And we 371 00:13:16,044 --> 00:13:17,504 have this sort of 372 00:13:17,964 --> 00:13:20,384 graph model in our minds. Right? We have, 373 00:13:21,245 --> 00:13:22,924 this big graph model in our mind of 374 00:13:22,924 --> 00:13:25,004 of the elements in our network and and 375 00:13:25,004 --> 00:13:26,924 the relationships between them. And so when we're 376 00:13:26,924 --> 00:13:29,350 thinking about a problem, you know, we're we're 377 00:13:29,350 --> 00:13:30,169 we're traversing 378 00:13:30,709 --> 00:13:32,570 that graph in our mind. And when we're 379 00:13:32,949 --> 00:13:35,429 typing command when we're gathering information about the 380 00:13:35,429 --> 00:13:37,350 state of the network, it's, the most common 381 00:13:37,350 --> 00:13:37,850 workflow 382 00:13:38,230 --> 00:13:39,829 for this would be I'm gonna give an 383 00:13:39,829 --> 00:13:41,625 example of, of how this 384 00:13:42,184 --> 00:13:44,664 I'm I'm calling it referential traversal as you 385 00:13:44,664 --> 00:13:46,504 go through this graph in your mind. When 386 00:13:46,504 --> 00:13:48,024 you're trying to find an endpoint in the 387 00:13:48,024 --> 00:13:50,044 network and you have an IP address, so 388 00:13:50,184 --> 00:13:51,784 you start in the middle and you type 389 00:13:51,784 --> 00:13:53,304 in show IP route and you get a 390 00:13:53,304 --> 00:13:55,784 next stop. Right? You you you see the, 391 00:13:55,945 --> 00:13:57,120 what the next stop is. 392 00:13:57,919 --> 00:13:58,659 And then 393 00:13:58,960 --> 00:14:01,360 you do a, you know, show LLDP and 394 00:14:01,360 --> 00:14:03,460 see if you can see the a router 395 00:14:03,839 --> 00:14:05,919 that's tied to that interface out that next 396 00:14:05,919 --> 00:14:08,240 hop. And then you SSH over to that 397 00:14:08,240 --> 00:14:10,240 router. And then you type in show IP 398 00:14:10,240 --> 00:14:11,460 route. And that might be 399 00:14:11,784 --> 00:14:14,024 a router where the IRB interface, the layer 400 00:14:14,024 --> 00:14:14,764 three interfaces. 401 00:14:15,384 --> 00:14:17,225 So, you do a show IP route, now 402 00:14:17,225 --> 00:14:19,304 it's connected. Right? And it's an IRB interface. 403 00:14:19,304 --> 00:14:22,264 So now you type in show ARP. Right? 404 00:14:22,264 --> 00:14:23,625 And you do a show ARP for the 405 00:14:23,625 --> 00:14:25,225 IP address of the thing you're looking for, 406 00:14:25,225 --> 00:14:26,605 and now you get a MAC address. 407 00:14:27,549 --> 00:14:29,309 And now you have the MAC address, and 408 00:14:29,309 --> 00:14:31,709 so you're you type in show, you know, 409 00:14:31,709 --> 00:14:33,309 you start looking for that MAC address in 410 00:14:33,309 --> 00:14:34,450 tables, and then eventually, 411 00:14:34,829 --> 00:14:37,230 you know, you might go another another hop 412 00:14:37,230 --> 00:14:38,690 or in a switch now, 413 00:14:39,070 --> 00:14:41,065 the next switch over. And then, 414 00:14:41,445 --> 00:14:42,825 you you figure out 415 00:14:43,205 --> 00:14:44,024 where that, 416 00:14:44,644 --> 00:14:46,485 MAC address is is attached to the edge 417 00:14:46,485 --> 00:14:48,004 of your network. And so now you found 418 00:14:48,004 --> 00:14:50,164 where that thing is. Well, you've you started 419 00:14:50,164 --> 00:14:52,245 off with, like, one piece of information that 420 00:14:52,245 --> 00:14:54,264 got transformed into another piece of information, 421 00:14:55,100 --> 00:14:57,339 which is the, you know, the MAC address 422 00:14:57,339 --> 00:14:59,360 and then that MAC address translated into 423 00:14:59,820 --> 00:15:02,539 an interface ID and a host name. So 424 00:15:02,539 --> 00:15:02,779 you've 425 00:15:03,419 --> 00:15:05,820 and and I call that referential traversal. You're 426 00:15:05,820 --> 00:15:07,634 using the last piece piece of information to 427 00:15:07,634 --> 00:15:09,235 look up the next piece of information to 428 00:15:09,235 --> 00:15:10,695 look up the next piece of information. 429 00:15:11,235 --> 00:15:13,095 And that's a pretty common that's 430 00:15:14,514 --> 00:15:16,455 90% of what we do as engineers, 431 00:15:16,834 --> 00:15:17,894 as network engineers. 432 00:15:18,274 --> 00:15:20,914 You know, when we're troubleshooting a network or 433 00:15:20,914 --> 00:15:22,054 we're thinking about 434 00:15:22,990 --> 00:15:24,990 not not just troubleshooting, but when we're thinking 435 00:15:24,990 --> 00:15:26,669 about a design change or something like that, 436 00:15:26,669 --> 00:15:27,889 and we're trying to model 437 00:15:28,350 --> 00:15:30,990 what we need to do, we're we're filling 438 00:15:30,990 --> 00:15:32,509 that graph out in our mind, and we're 439 00:15:32,669 --> 00:15:34,850 and the commands we type allow us to, 440 00:15:35,629 --> 00:15:36,610 get an accurate 441 00:15:37,764 --> 00:15:40,004 real time sort of view of of of 442 00:15:40,004 --> 00:15:41,925 the state of the network relative to that 443 00:15:41,925 --> 00:15:42,425 problem. 444 00:15:44,085 --> 00:15:46,085 So that's to me, that's what we ought 445 00:15:46,085 --> 00:15:48,165 to be focusing on for for automation. We 446 00:15:48,165 --> 00:15:50,184 should maybe because that is 447 00:15:50,485 --> 00:15:50,985 again, 448 00:15:51,570 --> 00:15:54,290 your ability to model, whether you're troubleshooting or 449 00:15:54,290 --> 00:15:56,370 doing something else, is a direct reflection of 450 00:15:56,370 --> 00:15:59,029 your experience and expertise. And 451 00:15:59,570 --> 00:16:01,250 it's the one thing, if you do very, 452 00:16:01,250 --> 00:16:01,990 very well, 453 00:16:02,370 --> 00:16:04,129 it makes it very easy to make the 454 00:16:04,129 --> 00:16:05,134 right decisions, 455 00:16:05,995 --> 00:16:06,815 design choices, 456 00:16:07,274 --> 00:16:08,654 configuration changes, etcetera, 457 00:16:09,675 --> 00:16:10,975 remedial remediation, 458 00:16:12,075 --> 00:16:12,575 changes. 459 00:16:13,274 --> 00:16:14,715 It it makes you it it gives you 460 00:16:14,715 --> 00:16:16,014 the ability to make 461 00:16:16,875 --> 00:16:19,595 the right choices. Right? In good, informed choices 462 00:16:19,595 --> 00:16:21,169 and not, you know, 463 00:16:22,029 --> 00:16:24,110 doing it terribly wrong and blowing things up, 464 00:16:24,110 --> 00:16:24,690 I guess. 465 00:16:25,710 --> 00:16:27,950 That's that's actually a really interesting way of 466 00:16:27,950 --> 00:16:30,029 looking at it because I've always said I've 467 00:16:30,029 --> 00:16:33,070 always believed that modeling is if I were 468 00:16:33,070 --> 00:16:35,149 somebody coming into this field, when I tell 469 00:16:35,149 --> 00:16:36,995 people, I tell them to read a lot, 470 00:16:36,995 --> 00:16:38,035 and they're like, why should I read a 471 00:16:38,035 --> 00:16:39,875 lot? Because you've gotta learn to model. I 472 00:16:39,875 --> 00:16:41,095 mean, modeling is 473 00:16:41,475 --> 00:16:42,455 the whole key 474 00:16:42,995 --> 00:16:45,154 to being a good network engineer. It just 475 00:16:45,154 --> 00:16:46,934 is. If you can't do it, then 476 00:16:47,394 --> 00:16:48,915 you just might as well give up because 477 00:16:48,915 --> 00:16:50,295 you're not gonna get anywhere. 478 00:16:50,920 --> 00:16:53,320 So what's the so that that modeling exercise 479 00:16:53,320 --> 00:16:54,840 where you show route and then you show 480 00:16:54,840 --> 00:16:56,360 up and you traverse the layer three and 481 00:16:56,360 --> 00:16:57,179 the layer two, 482 00:16:57,800 --> 00:16:59,399 a human does it at a certain speed. 483 00:16:59,399 --> 00:17:01,480 They're not that many humans, maybe none, that 484 00:17:01,480 --> 00:17:03,000 can do that at the speed that a 485 00:17:03,000 --> 00:17:05,275 computer can do it at. So what would 486 00:17:05,275 --> 00:17:07,994 you say is the consequence of giving that 487 00:17:07,994 --> 00:17:09,914 work over to a computer? What is it? 488 00:17:09,914 --> 00:17:11,515 I mean, it's important. The network engineer needs 489 00:17:11,515 --> 00:17:13,115 to do that, but if a network engineer 490 00:17:13,115 --> 00:17:15,195 doesn't have to bash key, you know, keys 491 00:17:15,195 --> 00:17:17,879 on the keyboard anymore, what what's the what 492 00:17:17,879 --> 00:17:19,480 is where does that bring us to? What's 493 00:17:19,480 --> 00:17:21,500 the consequence of that for network engineers? 494 00:17:22,039 --> 00:17:23,419 Yeah. Well, I don't think, 495 00:17:23,720 --> 00:17:25,879 I don't think automation is going to completely 496 00:17:25,879 --> 00:17:27,879 eliminate the need for people to to be 497 00:17:27,879 --> 00:17:29,740 at the keyboard, but what it can do 498 00:17:30,144 --> 00:17:31,984 is, look, it takes time to do that. 499 00:17:31,984 --> 00:17:33,585 Right? And all you need is the model. 500 00:17:33,585 --> 00:17:35,505 Right? You just need to understand the state 501 00:17:35,505 --> 00:17:36,085 of something. 502 00:17:36,384 --> 00:17:38,865 Like, that that's all you need. If you 503 00:17:38,865 --> 00:17:40,545 gotta log in to two or three or 504 00:17:40,545 --> 00:17:41,525 four different devices 505 00:17:41,904 --> 00:17:43,825 to figure out what's happening in a VERP 506 00:17:43,825 --> 00:17:44,325 domain, 507 00:17:44,990 --> 00:17:46,130 VRRP domain. 508 00:17:46,750 --> 00:17:48,910 You know, and and you don't and you 509 00:17:48,910 --> 00:17:49,809 don't do it right, 510 00:17:50,350 --> 00:17:51,090 you know, 511 00:17:51,789 --> 00:17:53,710 you're gonna you're gonna take time doing it, 512 00:17:53,710 --> 00:17:55,789 you're gonna potentially make mistakes, or you could 513 00:17:55,789 --> 00:17:56,610 have an automation 514 00:17:56,934 --> 00:17:58,695 that collects all the information and presents it 515 00:17:58,695 --> 00:18:01,015 to you in an easily consumable way. So 516 00:18:01,015 --> 00:18:02,154 you understand immediately, 517 00:18:02,455 --> 00:18:03,994 you know, you have a a good, 518 00:18:04,855 --> 00:18:07,174 immediate working model of of what's happening in 519 00:18:07,174 --> 00:18:09,869 that verb domain versus you trying to to 520 00:18:09,869 --> 00:18:11,710 build that model yourself. And at the time 521 00:18:11,710 --> 00:18:13,150 and the and you know what? Even if 522 00:18:13,150 --> 00:18:15,809 you're a CCIE or a JNCIE or whatever, 523 00:18:16,349 --> 00:18:18,190 I I know I I had two CCIEs, 524 00:18:18,190 --> 00:18:19,089 I had a JNCIE. 525 00:18:19,390 --> 00:18:21,390 I I all I still made mistakes all 526 00:18:21,390 --> 00:18:23,244 the time. Like, I wasn't, you know, you 527 00:18:23,244 --> 00:18:25,164 you forget or you don't or you overlook 528 00:18:25,164 --> 00:18:27,005 something and then you're, you know, you could 529 00:18:27,005 --> 00:18:28,845 waste two or three days worth of time 530 00:18:28,845 --> 00:18:31,964 based on, some bad understanding because you did 531 00:18:31,964 --> 00:18:33,644 something wrong, and now you got you you've 532 00:18:33,644 --> 00:18:35,900 lost that time. So the consequence is actually 533 00:18:35,900 --> 00:18:37,900 very positive for network engineers. It makes them 534 00:18:37,900 --> 00:18:38,559 more effective. 535 00:18:39,180 --> 00:18:42,460 And, and I think it it'll ultimately, it'll 536 00:18:42,460 --> 00:18:46,079 help right? Network engineers are not very 537 00:18:46,460 --> 00:18:48,079 well trusted by their organizations. 538 00:18:48,380 --> 00:18:50,755 They may they they they might believe they 539 00:18:50,755 --> 00:18:53,095 are, but if you I don't know. Any 540 00:18:53,234 --> 00:18:55,075 place that would look at a network engineer 541 00:18:55,075 --> 00:18:56,434 and say, you know what? I know it's 542 00:18:56,434 --> 00:18:58,835 10:00 on Monday, but go ahead and make 543 00:18:58,835 --> 00:19:00,615 those changes in the core of the network. 544 00:19:00,994 --> 00:19:03,234 It's like that this isn't Yeah. Yeah. Not 545 00:19:03,234 --> 00:19:05,414 gonna happen. So I think, you know, 546 00:19:05,769 --> 00:19:08,569 it'll make them more effective, and, ultimately, it'll 547 00:19:08,569 --> 00:19:10,750 it'll help build trust better. It'll it'll it'll 548 00:19:10,890 --> 00:19:13,049 improve the relationship network engineers have with the 549 00:19:13,049 --> 00:19:13,549 organization. 550 00:19:15,289 --> 00:19:17,289 So, essentially, your argument is we need to 551 00:19:17,289 --> 00:19:19,390 think how to automate 552 00:19:20,115 --> 00:19:23,234 what network engineers actually do, which is day 553 00:19:23,234 --> 00:19:23,894 to day 554 00:19:24,595 --> 00:19:27,315 just this traversal process of understanding how the 555 00:19:27,315 --> 00:19:28,214 network works, 556 00:19:28,674 --> 00:19:31,015 finding things in the network, things like that. 557 00:19:32,549 --> 00:19:34,789 Yeah. Yeah. And I'm not saying that configuration 558 00:19:34,789 --> 00:19:36,870 is important or we or we can't automate 559 00:19:36,870 --> 00:19:37,529 those things, 560 00:19:37,990 --> 00:19:40,150 but we have lots of tools already for 561 00:19:40,150 --> 00:19:40,650 doing 562 00:19:40,950 --> 00:19:43,670 automating configuration changes. Right? We can feed Yeah. 563 00:19:43,670 --> 00:19:45,994 There's now you can feed spreadsheets into Ansible 564 00:19:45,994 --> 00:19:47,835 so we can configure VLANs all over your 565 00:19:47,835 --> 00:19:48,734 network. I mean, 566 00:19:49,355 --> 00:19:51,755 we have lots of tooling for that, but 567 00:19:51,755 --> 00:19:54,474 we have almost no tooling for look, most 568 00:19:54,474 --> 00:19:57,034 network engineers can only make those changes, during 569 00:19:57,034 --> 00:19:58,894 a short window of time during the week. 570 00:19:59,700 --> 00:20:02,099 Usually, at at the most inconvenient time, like, 571 00:20:02,099 --> 00:20:04,359 Sunday at, like, two or 3AM, 572 00:20:05,460 --> 00:20:06,759 they're making those changes. 573 00:20:07,380 --> 00:20:08,579 And then the rest of the week, what 574 00:20:08,579 --> 00:20:11,380 are they doing? You know, if I you 575 00:20:11,380 --> 00:20:12,525 know, we ought to be 576 00:20:12,924 --> 00:20:14,765 focusing on what they're doing in the rest 577 00:20:14,765 --> 00:20:15,424 of the week, 578 00:20:15,964 --> 00:20:16,545 I think. 579 00:20:18,045 --> 00:20:20,125 Yeah. I like that, actually. Yeah. And, again, 580 00:20:20,125 --> 00:20:21,265 I go back to this. 581 00:20:22,285 --> 00:20:25,085 We don't really think a lot about what 582 00:20:25,085 --> 00:20:25,825 it is, 583 00:20:26,390 --> 00:20:28,549 like you said, what network engineers are actually 584 00:20:28,549 --> 00:20:30,470 doing. And a lot of it is around 585 00:20:30,470 --> 00:20:33,829 modeling. So tools that help you understand the 586 00:20:33,829 --> 00:20:35,450 way the network actually works 587 00:20:36,149 --> 00:20:37,909 and model the way things are. Now this 588 00:20:37,909 --> 00:20:39,289 goes back as well to, 589 00:20:40,404 --> 00:20:42,484 we have a serious problem in the networking 590 00:20:42,484 --> 00:20:44,484 industry, and maybe this view of, 591 00:20:45,125 --> 00:20:47,545 maybe this view of model driven 592 00:20:48,164 --> 00:20:50,345 automation is a big deal here because 593 00:20:50,644 --> 00:20:51,765 one of the big problems I think we 594 00:20:51,765 --> 00:20:53,205 have in the networking industry is we've know 595 00:20:53,285 --> 00:20:54,565 we have a lot of people who don't 596 00:20:54,565 --> 00:20:55,305 know networking, 597 00:20:56,380 --> 00:20:58,380 And they're network engineers. I mean, I'm I'm 598 00:20:58,380 --> 00:21:00,619 not being, I'm not trying to be, you 599 00:21:00,619 --> 00:21:01,119 know, 600 00:21:01,420 --> 00:21:03,420 it's just the truth. We have a ton 601 00:21:03,420 --> 00:21:04,079 of people 602 00:21:04,619 --> 00:21:07,839 who understand how to configure things and understand 603 00:21:08,299 --> 00:21:10,565 if I take this this product and I 604 00:21:10,565 --> 00:21:12,325 put it with that product, I can make 605 00:21:12,325 --> 00:21:14,085 this product do that. And they understand the 606 00:21:14,085 --> 00:21:16,565 products really well. But they just don't understand 607 00:21:16,565 --> 00:21:18,805 the network at all. They just totally they're 608 00:21:18,805 --> 00:21:20,825 lost when it comes to the network itself. 609 00:21:22,820 --> 00:21:25,619 So I think maybe thinking about this could 610 00:21:25,619 --> 00:21:27,799 actually be a way, a wedge to say, 611 00:21:27,859 --> 00:21:29,220 we need to actually think about how the 612 00:21:29,220 --> 00:21:31,299 network works. We need to think about how 613 00:21:31,299 --> 00:21:33,059 the network. We we need to talk about 614 00:21:33,059 --> 00:21:35,335 traversing this and saying, oh, I'm gonna traverse 615 00:21:35,494 --> 00:21:36,934 I'm gonna do a show IP route here 616 00:21:36,934 --> 00:21:38,234 and do this and do that. 617 00:21:38,774 --> 00:21:40,375 Alright. But what you've got to think about 618 00:21:40,375 --> 00:21:40,875 is 619 00:21:41,494 --> 00:21:43,255 when you look at that, what you're actually 620 00:21:43,255 --> 00:21:45,654 doing is you're thinking about how does the 621 00:21:45,654 --> 00:21:46,554 network converge? 622 00:21:46,960 --> 00:21:48,820 How is how is that about advertised? 623 00:21:49,680 --> 00:21:52,080 How am I getting this information? Where does 624 00:21:52,080 --> 00:21:53,299 this come from? 625 00:21:53,759 --> 00:21:56,160 This is not configuration level stuff. This is 626 00:21:56,160 --> 00:21:57,680 how does it actually work? This is not 627 00:21:57,680 --> 00:21:59,355 even packet level stuff. And by the way, 628 00:21:59,674 --> 00:22:00,954 I know lots of people who know the 629 00:22:00,954 --> 00:22:02,714 packet level details really, really well, and they 630 00:22:02,714 --> 00:22:04,394 know how to configure it, and they still 631 00:22:04,394 --> 00:22:06,234 can't tell me how BGP converges on a 632 00:22:06,234 --> 00:22:09,535 Clove fabric. They still know. They're totally lost. 633 00:22:11,515 --> 00:22:12,954 I don't know. If you asked me that 634 00:22:12,954 --> 00:22:14,794 question right now, I probably fell at answering 635 00:22:14,794 --> 00:22:15,089 it. 636 00:22:17,009 --> 00:22:18,609 I, well, but this is a, this is 637 00:22:18,609 --> 00:22:20,369 a really good point. I, and a lot 638 00:22:20,369 --> 00:22:21,990 of people don't, you know, 639 00:22:22,529 --> 00:22:24,769 they don't understand just how much they don't 640 00:22:24,769 --> 00:22:26,869 know. I think one of the most humbling 641 00:22:26,929 --> 00:22:29,009 experiences I've ever had was working for Dave 642 00:22:29,009 --> 00:22:29,909 Meyer at Brocade, 643 00:22:30,265 --> 00:22:31,964 and going to an IETF meeting 644 00:22:32,265 --> 00:22:34,365 because I thought, Hey, you know, I'm, I'm, 645 00:22:34,825 --> 00:22:37,065 I've had these certifications. I've read different, you 646 00:22:37,065 --> 00:22:37,964 know, George, 647 00:22:38,585 --> 00:22:39,964 Voorhees' book on 648 00:22:40,265 --> 00:22:42,684 network algorithmics. You know, I've have, I have 649 00:22:42,744 --> 00:22:44,765 some different books like that that are quasi 650 00:22:44,825 --> 00:22:45,779 comp sci level, 651 00:22:46,500 --> 00:22:47,480 stuff about 652 00:22:47,940 --> 00:22:49,000 networking algorithms. 653 00:22:49,539 --> 00:22:50,039 And, 654 00:22:51,299 --> 00:22:53,059 and then I go to I start working 655 00:22:53,059 --> 00:22:54,740 for Dave, and I go to an ITF 656 00:22:54,740 --> 00:22:56,580 meeting, and I'm, like, my head is melting. 657 00:22:56,580 --> 00:22:58,359 Like, oh my gosh. I had 658 00:23:00,365 --> 00:23:02,605 maybe the dumbest person in Prague right now 659 00:23:02,605 --> 00:23:04,365 when I was I was in Prague at 660 00:23:04,365 --> 00:23:05,025 the ITF. 661 00:23:06,125 --> 00:23:08,444 People gave them arguments about like K fan 662 00:23:08,444 --> 00:23:10,605 out and stuff with multicast and like, you 663 00:23:10,605 --> 00:23:11,105 know, 664 00:23:11,804 --> 00:23:13,325 giving pulling out a sheet of paper and 665 00:23:13,325 --> 00:23:16,080 writing down, you know, like math and, and 666 00:23:16,080 --> 00:23:16,580 arguing 667 00:23:17,200 --> 00:23:18,420 about multicast. And 668 00:23:18,720 --> 00:23:20,660 I've never thought about it at that level, 669 00:23:21,680 --> 00:23:23,680 because I don't, I don't design protocols. And 670 00:23:23,680 --> 00:23:25,140 you know what, I think, 671 00:23:25,680 --> 00:23:28,240 I do think if more people understood that 672 00:23:28,240 --> 00:23:28,640 stuff, 673 00:23:29,119 --> 00:23:30,019 our, we, 674 00:23:30,615 --> 00:23:32,375 our field would be in much better shape 675 00:23:32,375 --> 00:23:34,454 than it is now. We, we've, our knowledge 676 00:23:34,454 --> 00:23:37,654 has been driven by certifications that have dollar 677 00:23:37,654 --> 00:23:40,454 value tied to them. So where do we 678 00:23:40,454 --> 00:23:42,214 put our brain cycles and what knowledge do 679 00:23:42,214 --> 00:23:44,075 we ingest? We ingest the thing 680 00:23:44,375 --> 00:23:45,914 that has the biggest payoff, 681 00:23:47,549 --> 00:23:49,950 or a most immediate payoff. Right? Not the 682 00:23:49,950 --> 00:23:52,589 thing that probably actually helps us serve the 683 00:23:52,589 --> 00:23:55,069 business better. Yep. I think I think that's 684 00:23:55,069 --> 00:23:57,730 absolutely true. Yeah. I think that's absolutely true. 685 00:23:57,869 --> 00:23:59,894 And so, I mean, it's a good thing. 686 00:23:59,894 --> 00:24:01,414 Certifications are a good thing, and they're a 687 00:24:01,414 --> 00:24:03,575 bad thing. I mean, everything is trade offs. 688 00:24:03,575 --> 00:24:05,174 And I always say this, if you can't 689 00:24:05,174 --> 00:24:06,694 find the trade offs, then you haven't looked 690 00:24:06,694 --> 00:24:08,454 hard enough. Because I think that's just the 691 00:24:08,454 --> 00:24:10,534 way it is, and we often take things 692 00:24:10,534 --> 00:24:12,134 as absolute in our field, and we don't 693 00:24:12,134 --> 00:24:13,654 think about the trade offs and what they 694 00:24:13,654 --> 00:24:16,410 actually mean and, how how you work around 695 00:24:16,410 --> 00:24:16,910 that. 696 00:24:17,289 --> 00:24:19,210 And and I think that also as well 697 00:24:19,210 --> 00:24:21,609 that we we do like you said, we 698 00:24:21,609 --> 00:24:23,849 ingest what we think will make us more 699 00:24:23,849 --> 00:24:24,910 valuable tomorrow, 700 00:24:25,369 --> 00:24:26,349 and the business 701 00:24:27,065 --> 00:24:29,785 wants the network to run, so we think 702 00:24:29,785 --> 00:24:32,184 what makes the network run is knowing how 703 00:24:32,184 --> 00:24:33,325 to configure hardware 704 00:24:34,105 --> 00:24:36,424 and software. That's that's what it comes down 705 00:24:36,424 --> 00:24:38,285 to. And that's not really, 706 00:24:38,904 --> 00:24:40,424 and that also leads us down the whole 707 00:24:40,424 --> 00:24:42,630 complexity path. Right? Oh, if I could just 708 00:24:42,630 --> 00:24:43,930 get this nerd knob, 709 00:24:44,309 --> 00:24:45,910 then I could solve that problem and the 710 00:24:45,910 --> 00:24:47,289 network would just work. 711 00:24:47,670 --> 00:24:49,930 No. Yeah. No. No. No. No. It won't. 712 00:24:50,230 --> 00:24:50,970 Trust me. 713 00:24:51,830 --> 00:24:53,590 Trust me. I've coded a few of those 714 00:24:53,590 --> 00:24:55,634 nerd knobs. You actually don't want them. 715 00:24:57,474 --> 00:24:59,154 Yeah. That's true. I, 716 00:24:59,634 --> 00:25:01,474 there's a there's a really great book called 717 00:25:01,474 --> 00:25:03,015 MPLS enabled applications, 718 00:25:03,555 --> 00:25:05,234 that that I think they published through. Oh, 719 00:25:05,234 --> 00:25:07,234 yeah. Yeah. Yeah. Like, three editions of it. 720 00:25:07,234 --> 00:25:07,734 Right? 721 00:25:08,630 --> 00:25:10,390 What I really love about that book is 722 00:25:10,390 --> 00:25:10,970 it was, 723 00:25:11,509 --> 00:25:14,009 if if, if you haven't read it is 724 00:25:14,230 --> 00:25:17,029 they they go into they explain in Junos 725 00:25:17,029 --> 00:25:19,109 where where some of these nerd knobs came 726 00:25:19,109 --> 00:25:20,250 from and the very 727 00:25:20,695 --> 00:25:23,015 specific network designs and use cases they were 728 00:25:23,015 --> 00:25:25,654 they were meant for. Right. And, and why 729 00:25:25,654 --> 00:25:27,494 you wouldn't wanna use them if you don't 730 00:25:27,494 --> 00:25:29,095 have that network design because it can, you 731 00:25:29,095 --> 00:25:31,335 know, it's gonna cause convergence problems, it's gonna 732 00:25:31,335 --> 00:25:31,835 cause, 733 00:25:33,000 --> 00:25:36,279 whatever. You know, all memory, CPU problems, all 734 00:25:36,279 --> 00:25:37,019 kinds of things. 735 00:25:37,480 --> 00:25:37,980 And, 736 00:25:38,519 --> 00:25:39,000 and and 737 00:25:39,440 --> 00:25:41,880 yeah. The if only I could filter out, 738 00:25:41,880 --> 00:25:44,200 you know, the, the BGP option that says 739 00:25:44,200 --> 00:25:45,825 this route went through a route reflector, then 740 00:25:45,825 --> 00:25:47,525 then I can make my network do anything, 741 00:25:47,664 --> 00:25:48,565 including exploding. 742 00:25:49,105 --> 00:25:50,244 It's like opening up, 743 00:25:50,944 --> 00:25:51,684 for sure. 744 00:25:52,144 --> 00:25:53,744 And then it explodes, and you're up at 745 00:25:53,744 --> 00:25:55,345 02:00 in the morning trying to fix it 746 00:25:55,345 --> 00:25:57,184 and complaining about how stressful your job is. 747 00:25:57,184 --> 00:25:57,684 So 748 00:26:05,029 --> 00:26:07,589 But the complexity thing is is interesting because, 749 00:26:07,910 --> 00:26:09,750 people have no scale of reference for what 750 00:26:09,750 --> 00:26:11,609 is complex and what is simple, 751 00:26:12,230 --> 00:26:14,434 in in networking. That's another thing that I 752 00:26:14,434 --> 00:26:16,615 I find people they throw those words around, 753 00:26:18,115 --> 00:26:18,914 as as, 754 00:26:19,954 --> 00:26:22,835 sort of to win arguments. Right? Mhmm. Well, 755 00:26:22,835 --> 00:26:24,515 I don't use MPLS because I like to 756 00:26:24,515 --> 00:26:25,974 keep it simple. So 757 00:26:26,349 --> 00:26:28,349 what I do is, you know, and instead 758 00:26:28,349 --> 00:26:30,109 of just using a VRF, what they do 759 00:26:30,109 --> 00:26:31,869 is they build a network and put a 760 00:26:31,869 --> 00:26:33,629 firewall on it that NATs, and they do 761 00:26:33,629 --> 00:26:35,089 that. And everywhere 762 00:26:35,470 --> 00:26:36,990 around their network, and then they have a 763 00:26:36,990 --> 00:26:39,825 core, and they're like, see, super simple. And 764 00:26:40,485 --> 00:26:40,985 then 765 00:26:41,644 --> 00:26:43,644 in three years, no, okay, so you have 766 00:26:43,644 --> 00:26:45,664 no dynamic routing beyond, you know, between 767 00:26:46,125 --> 00:26:46,865 going into 768 00:26:47,164 --> 00:26:49,825 throughout your network. And second, you know, 769 00:26:50,365 --> 00:26:52,525 in two or three years, almost guaranteed every 770 00:26:52,525 --> 00:26:54,365 time they have permit any any on all 771 00:26:54,365 --> 00:26:55,025 those firewalls. 772 00:26:55,380 --> 00:26:57,859 Yep. So basically, you have a statically routed 773 00:26:57,859 --> 00:26:58,359 node 774 00:26:59,380 --> 00:27:00,200 sitting in 775 00:27:00,900 --> 00:27:03,299 all around your network because you wanted to 776 00:27:03,299 --> 00:27:04,900 keep it simple. Right? And by the way, 777 00:27:04,900 --> 00:27:07,380 all those firewalls are clustered and the clustering 778 00:27:07,380 --> 00:27:09,424 fails all the time. Glad, super glad we 779 00:27:09,424 --> 00:27:11,944 can. But but it's okay to make the 780 00:27:11,944 --> 00:27:14,204 configuration of all those static routes. 781 00:27:16,184 --> 00:27:16,684 Yeah. 782 00:27:20,585 --> 00:27:23,065 I I think some people use complexity as 783 00:27:23,065 --> 00:27:25,359 a kind of a pseudonym for I don't 784 00:27:25,359 --> 00:27:27,679 understand or I don't I don't fully grasp 785 00:27:27,679 --> 00:27:29,599 it. It's complex. I don't I don't wanna 786 00:27:29,599 --> 00:27:31,440 go there. I don't know. Or I don't 787 00:27:31,440 --> 00:27:33,200 know it. I don't know how to configure 788 00:27:33,200 --> 00:27:36,159 that. So therefore, it's complex. Mhmm. Right. Yeah. 789 00:27:36,159 --> 00:27:37,679 I think I think that's I think that's 790 00:27:37,679 --> 00:27:38,900 a very common thing. 791 00:27:39,279 --> 00:27:40,259 Yeah. So 792 00:27:41,355 --> 00:27:42,654 alright. Cool. So 793 00:27:43,355 --> 00:27:45,115 what else you got, Derek? Anything else you 794 00:27:45,115 --> 00:27:47,035 wanna add to this before we close off 795 00:27:47,035 --> 00:27:48,795 this recording and bring you back on in 796 00:27:48,795 --> 00:27:50,394 the future? Because I know we're gonna have 797 00:27:50,394 --> 00:27:51,835 to bring you back on now because this 798 00:27:51,835 --> 00:27:52,974 has been a great show. 799 00:27:53,380 --> 00:27:56,019 Yeah, thanks. Yeah, it was fun. No, just 800 00:27:56,019 --> 00:27:57,140 thanks for having me. And, 801 00:27:57,940 --> 00:28:00,740 you know, I'm gonna try to combine these, 802 00:28:01,700 --> 00:28:02,200 these, 803 00:28:02,660 --> 00:28:04,819 these blog posts into there's several of them, 804 00:28:04,819 --> 00:28:06,259 and I think they can all be fit 805 00:28:06,259 --> 00:28:07,720 together into one longer, 806 00:28:08,144 --> 00:28:10,565 maybe, white papers thing or something like that. 807 00:28:10,865 --> 00:28:13,184 And we'll publish them somewhere. Yeah. Yeah. Yeah. 808 00:28:13,184 --> 00:28:14,945 Cool. And tell me, mate, and I'll put 809 00:28:14,945 --> 00:28:16,144 them in the show notes when you do 810 00:28:16,144 --> 00:28:18,465 so. So I just also wanna say, 811 00:28:18,945 --> 00:28:21,184 for people who wanna understand how networks work, 812 00:28:21,184 --> 00:28:22,085 I have, 813 00:28:22,809 --> 00:28:24,570 I'm gonna I'm gonna do something that seems 814 00:28:24,570 --> 00:28:26,109 cheesy, but I I gotta say, 815 00:28:26,410 --> 00:28:28,730 your book that you wrote that, you you 816 00:28:28,730 --> 00:28:31,049 told me to to look through was is 817 00:28:31,049 --> 00:28:32,650 absolutely fantastic. That's 818 00:28:33,049 --> 00:28:35,225 it's Oh, thanks. It's a permanent thing on 819 00:28:35,225 --> 00:28:37,545 myself. The one you just wrote about, Problems 820 00:28:37,545 --> 00:28:39,164 and Solutions in Computer Networking. 821 00:28:39,545 --> 00:28:42,345 Yes. That's a fantastic book. I really think 822 00:28:42,345 --> 00:28:44,105 it's more of a mindset than it is 823 00:28:44,105 --> 00:28:46,585 what's in the book itself, honestly, in many 824 00:28:46,585 --> 00:28:48,285 ways. It's just thinking through, 825 00:28:48,750 --> 00:28:50,349 this is a problem. What am I what 826 00:28:50,349 --> 00:28:53,250 are my possible solutions? Right? And that's really 827 00:28:53,789 --> 00:28:54,990 I mean, I know it's a big book 828 00:28:54,990 --> 00:28:57,170 for just trying to say that, but I 829 00:28:57,390 --> 00:28:57,890 think 830 00:28:58,990 --> 00:29:01,170 that's I think that's the bottom line. So 831 00:29:01,869 --> 00:29:04,049 so, Tom, you're not blogging right now. Right? 832 00:29:04,349 --> 00:29:05,384 No. No. Not at the 833 00:29:05,765 --> 00:29:07,684 moment. Yeah. We're gonna fix that eventually, but 834 00:29:07,684 --> 00:29:09,684 it's okay. And I know Derek doesn't blog 835 00:29:09,684 --> 00:29:11,605 right now, but you're on LinkedIn. Right, Derek? 836 00:29:11,605 --> 00:29:12,105 Twitter? 837 00:29:12,484 --> 00:29:13,865 Twitter. On Twitter. 838 00:29:15,204 --> 00:29:17,384 Yeah. I'm I'm on Twitter. I'm at CloudToad. 839 00:29:18,529 --> 00:29:20,769 And, right now my, my big thing in 840 00:29:20,769 --> 00:29:23,089 life is an open source project called NRE 841 00:29:23,089 --> 00:29:25,410 Labs, that you can also follow on Twitter 842 00:29:25,410 --> 00:29:27,990 at NRE Labs. It's about network automation, 843 00:29:28,929 --> 00:29:31,190 and how we're trying to rebrand network automation 844 00:29:31,250 --> 00:29:33,525 as actually reliability engineering. 845 00:29:35,105 --> 00:29:37,345 So Which we should talk about that sometime, 846 00:29:37,345 --> 00:29:39,664 actually, because that's a really cool concept. It's 847 00:29:39,664 --> 00:29:42,105 just the concept of network reliability engineering 848 00:29:42,545 --> 00:29:44,325 Yeah. Is a really good one. Yeah. 849 00:29:44,630 --> 00:29:46,789 So good. Excellent. So in our labs, people 850 00:29:46,789 --> 00:29:48,070 can find you there and they can actually 851 00:29:48,070 --> 00:29:49,289 go to n r e labs 852 00:29:49,590 --> 00:29:52,970 and try out some of the open source 853 00:29:53,750 --> 00:29:56,549 stuff that you've been working on as far 854 00:29:56,549 --> 00:29:57,049 as 855 00:29:57,384 --> 00:30:00,345 like, just trying automation out, like it's out 856 00:30:00,345 --> 00:30:01,884 there. You can go try it. 857 00:30:02,265 --> 00:30:04,025 Yeah. Oh, yeah. It's really cool. You get 858 00:30:04,025 --> 00:30:07,085 your own interactive sandbox environment with real networking 859 00:30:07,144 --> 00:30:07,644 nodes. 860 00:30:08,105 --> 00:30:09,944 It's and it's marketing wall free. You don't 861 00:30:09,944 --> 00:30:11,704 have to give your email, your name, anything 862 00:30:11,704 --> 00:30:12,880 like that. You can just click on a 863 00:30:12,880 --> 00:30:14,880 lesson, and then in your browser, you get 864 00:30:14,880 --> 00:30:17,140 this sandbox environment and some instructional 865 00:30:17,679 --> 00:30:19,679 stuff for, you know, for a given automation 866 00:30:19,679 --> 00:30:20,179 topic. 867 00:30:20,799 --> 00:30:25,700 And the URL for that is labs.networkreliability.engineering. 868 00:30:26,315 --> 00:30:28,075 Okay. Cool. I'll put I'll try to remember 869 00:30:28,075 --> 00:30:29,615 to put it in the show notes too. 870 00:30:29,994 --> 00:30:31,755 So back to you, Tom. You're on Twitter. 871 00:30:31,755 --> 00:30:34,234 Right? Yeah. Twitter at Tom Ammon and then 872 00:30:34,234 --> 00:30:37,035 also on LinkedIn. Okay. Cool. And I'm Russ. 873 00:30:37,035 --> 00:30:38,474 You know, he's getting me at rule eleven 874 00:30:38,474 --> 00:30:40,875 dot tech on Twitter, routing geek. Although, I 875 00:30:40,875 --> 00:30:42,599 don't PM on Twitter for the most part. 876 00:30:42,599 --> 00:30:44,220 I really don't log in very much 877 00:30:44,599 --> 00:30:44,920 and, 878 00:30:45,400 --> 00:30:47,880 I'm also on Linkedin. You can always find 879 00:30:47,880 --> 00:30:49,640 me there. So that's cool And thanks for 880 00:30:49,640 --> 00:30:51,240 joining us for this episode of the hedge 881 00:30:51,240 --> 00:30:52,539 and we'll see you next time 882 00:31:13,829 --> 00:31:15,910 Thank you for joining us. You can find 883 00:31:15,910 --> 00:31:18,340 the hedge at rule11.tech.