#047: AutoCon0 Recap Insights with Danny Wade (DevNetDan)

Eric Chou: 0:00

Network Automation Nerds Podcast. Hi guys, this is Eric Cho. I hope you're having a fantastic day. I want to let you know this episode was originally a guest speaking session with Denny at our Network Animation Learning Community for his thoughts about the recently concluded Autocon Zero Conference. It is presented here in its original length of one hour, so a bit longer than the normal episodes. You would also hear other members voices in the Q&A session toward the end. If you find it useful and are interested in joining our next guest speaking session, you could sign up via community-signupnercanimationcommunity. The link is also in the show notes. Now back to the show. Yeah, without further ado, you know. Thanks for agreeing to be here, den.

Eric Chou: 0:57

Yeah no, thank you for inviting me and yeah, I appreciate being here. No, I mean this. This is not the first time. I mean first time you being a guest speaker. But this is not the first time you Would talk, for sure. And you're not the first time we have a conversation about network automation, is it?

Danny Wade: 1:16

right? No, not at all.

Eric Chou: 1:19

Yeah, so what do you like? How do you? How did you hear about the event?

Danny Wade: 1:24

Yeah, so I'll try to erase as many acronyms as possible. So, yes, please the blog article NAF, autocon Zero. Yeah, this is Network Automation Forum. I believe I first saw them on LinkedIn. I think that's where their presence like. I saw their presence first.

Eric Chou: 1:43

Yeah.

Danny Wade: 1:44

No. So you know it definitely piqued my interest, but you know, as you know, eric, like there are a lot of resources for network automation, so I wasn't sure exactly, like the purpose of it, whether it was a conference, whether it's training like or a mix, and so I kind of like, I just I followed their page and kind of sat back and you know, weeks went by and Started noticing they were getting a lot of sponsorships.

Eric Chou: 2:09

Right. So we're like how do I fill out that that expense report right, right, right Is the training Right.

Danny Wade: 2:17

Yeah. So yeah, I was trying to figure out, like, what's going on here, because the motto is like it's on their website, I'm sure yeah, basically let me pull out their website.

Danny Wade: 2:27

Yeah, it's right there. Why haven't we seen full adoption of network automation yet, which I thought I was like? You know that's a question I think everyone asked. That's a network automation, you know, why aren't we seeing? We're seeing change but we're not really seeing full adoption. So I thought it was an interesting, you know motto tagline. So you know I followed them and solve more as, as the program like, it seemed like their followers were growing, they had more sponsors. So, yeah, then I just decided I was like well, this, could you know this technically would be like a prevent, like a Cisco live, equivalent to a A conference, right, yeah, an industry conference.

Eric Chou: 3:08

So you know, that's when I decided to go but like Do you do you want to define like full adoption of network automation? Or because you know, I mean yeah, because that's written up in your blog post too, right, right?

Danny Wade: 3:24

Yeah, so network, I would say adoption would be operationalizing it. So I'll kind of expand from there. So operationalizing would be Replacing existing processes with new Automated, focused ones. Right, it doesn't mean it replaces everything. But for example, if you're simple example, you need to update the software on your network devices and you know, you have one guy, two guys on the team that might know Ansible or Python, and they have a couple scripts that work here and there. That's not really adoption. Adoption would be you go into, you know, go through the change process, and your implementation plan Doesn't include hey, we copy this to flash, we do that, we do that, um, you know, we rely on this automated process, this system, um, and then obviously, depending on the change management, they may want to know the the technicalities of it, but You're more or less pointing it to. Hey, we have an automation system, not just a one-off script on someone's laptop.

Eric Chou: 4:33

Yeah, yeah, I mean I think I think that's a pretty good Way of coming in from, like an operator, right, because I think the the vendor, depending on what they're selling, they also have, like different, like you know, definition on adoption, right, like a lot of vendors sells this, uh, one size fits all, or they try to tree your whole network as a Uniform fabric and they don't care, you know, which part gets taken out and put it in, and that that is a very different picture when you say like adopt that versus just like operationalize.

Danny Wade: 5:10

Yeah, I like to use the operationalizing as a good example, just because ops is kind of where everyone starts most of the time. Yeah, you don't normally just get thrown in from your ccna into an architectural so Rarely works out right right, so, but yeah, everyone starts an ops.

Danny Wade: 5:31

I like to use that example. But no, this automation there, network automation is just a small sub, you know, subsector, of infrastructure automation, right, so we will get into all that. But network, in network automation, you have the operation side, um. We have observability, um, there's plenty. There's other sectors, sectors, I'll call them, within network automation that you definitely can create examples, um, I mean observability. There was a whole panel and we could talk about that. But there was a whole panel at this conference that literally just talked about observability, um, and thinking about streaming telemetry and, you know, moving off of s and mp, right, and that whole conversation. So Um, that in itself right, is its own technically could be look at as adoption of automation, um. So there's just so many different pieces with automation. It's tough to just have a blanket statement like that, um. But yeah.

Danny Wade: 6:28

I hope it. I hope the point is I hope it just If you're listening that One of those examples makes sense. But that's why I like to start with operations, because I think a lot of people can understand that.

Eric Chou: 6:40

Yeah, and I think eddie also mentioned in the chat that you know what grabbed his attention initially was network right, like there's a lot of Conferences out there, but really there's one that's just networking focus, I mean set aside like network or or, uh, cisco, live right.

Danny Wade: 7:00

Um, yeah, exactly Um, and I guess that might be a good segue. And to start just like talking about the conference itself, Of course, yeah, go for it so and then you know, kind of starting off with you know, the highlight section of my blog, like they were vendors and you know I meant I mentioned sponsors. They're sponsors of the event and they had little boosts set up. You know a couple tables here and there, I don't know how many there were total. I think. I read somewhere somewhere in the 20s, 24, 26.

Eric Chou: 7:31

Yeah, it seems like they have different tiers. So on the website, if I scroll it down right here, so there's, you know I tension. There's uh Backbox which I've never heard of, but certainly you know, ip fabric, juniper selector, uh infinite, I can take net box, labs, network to code People or companies that you've heard of. And of course, arista, which I think you'll you'll go into too, right.

Danny Wade: 7:55

Right, yep, so I, yeah, and in the in my blog I mentioned a couple of those vendors and you know I stopped by a few of their Uh, their setups, but I just mentioned a couple. So, starting off, like Arista I mentioned, because I have a you know good friend, holy apparez works there. I've, I've, yeah, right, you know holy as well. So, yeah, I mean I, it was, uh, it was an interesting. It was interesting from that perspective, though, right, with the Vendors being there, because, unlike larger conferences, other tech conferences, where they're just trying to sell you a product, yeah, the conversations that I ever heard and were a part of while I was out there, what they were more constructive is the best way I could describe them.

Danny Wade: 8:40

You know it was like what problems are you seeing? And Some of them were just in general conversations about automation. It wasn't like, hey, this product, you should get my product because, xyz, it was just well, what problems are you having? Oh, have you thought about this? Have you tried this? Um, it was really Just like a technology driven conversation, um, versus product driven. So it was nice. It was kind of refreshing I guess the best way to put it to not be flooded with, uh, marketing emails and hey, you should buy this product because this, it was just this is what our product does. Take a test drive if you want, or if you want me to you know, kind of go into details, or 10 000 foot view, let me know. Um, so, definitely, you know. Shout out to the, the sponsors. They, I think all their people did a great job, kind of explaining their products without being too salesy.

Eric Chou: 9:36

Yeah, and I think Keith also mentioned, uh, you know Keith, who works for juniper in the chat, that juniper so yeah, kudos man. I mean, you know we'll probably like when we open up the audio session later on, we'll probably have couple minutes for his thoughts as well. But, um, but yeah, you know, thanks for all those sponsors. Like you know, you can't can happen without them. So I'm glad to hear that they're Not a lot of marketing and not a lot of, you know, like flops, right?

Danny Wade: 10:03

So people who are there probably are more very tech focused, as you mentioned right, yeah, so I'll quickly just touch on some of the um, the sponsors that I mentioned in the blog post. So rissa, as I mentioned, was there. Um, they had a demo with um, their avd product. So you know, using ansible, able to deploy a validated design um. And then I also got to talk with julio about their um, a network test automation framework they have called anta.

Eric Chou: 10:36

Is that like a Similar to py pts, or what's your, what's your take on it?

Danny Wade: 10:42

So I don't really have a take yet. I keep the documentation and I promised him that I would take a look at it. Um, but it's it seems very, it seems very easy to like a low entry barrier To getting started. He showed me some of the documentation so I did promise him I'll take a look at that, because network test automation that is my, that's what I've been enjoying doing, um, you know, currently. So, yeah, so that was a rissa.

Danny Wade: 11:13

Um, they just, like I said, it was very, it was very casual in a way, to where you could walk up, see a demo, talk to julio, talk to um, uh, daniel as well, he was there as a coworker To where they could go through a demo, ask about their products and so forth. Um, yeah, the next one you have up here no kia, no kia Keeps crushing it. I mean there's really nothing else to say. Like, no kia, uh, they had um container lab. They showed you know, the ins and outs of container lab. If you haven't heard of container lab, please, please, check it out. Um, it is one of the easiest tools. It's so refreshing to see you could basically simulate a network Just with containers and there are a lot of possibilities with it, um. So you know they went through Showed container lab. They talked about it. You know, at their booth Um, they also talked on stage actually that they had a representative from no kia on stage talked about they have a um, an ai.

Danny Wade: 12:22

It's like a chat gpt yeah, sr Linux, gpt, yeah, and so basically you can ask it's, it's built into the software, it's, you can install it in uh, sr Linux and you can ask, almost like a chat gpt interface. You could say In plain english like ask ai, how many bgp neighbors do I have? Um, or all my bgp neighbor relationships healthy or, you know, stable? So is some very, some things that those are very easy examples. Right, they also showed an example of if I wanted to configure bgp with four neighbors with these ip's, how would I do that? And it would print out you know your configuration for you and show you how to do it. Um.

Eric Chou: 13:05

It's where. Where do you install the Uh uh srl gpt? Do you install on the host machine that's running these containers, Correct? Or do you install them inside of the container?

Danny Wade: 13:17

Yes, on the device. So in the context of ContainerLab, yeah, if you had SR Linux, I'm assuming it works in their containerized image. But yes it's on the device.

Eric Chou: 13:29

Oh, ok, so like on the host machine, so it has visibility into the connections and so on. Correct.

Danny Wade: 13:36

Yeah, so on the router, on the switch, if it's in a container, in the container. Don't hold me to the container piece, but I believe actually what they showed on stage.

Eric Chou: 13:47

OK, anyway, yeah, ok. Well, we'll have more information, probably because this is still experimental, but I imagine if you need visibility into the connections between different hosts, then they have to have that macro level.

Danny Wade: 14:02

Yeah, the interesting play I saw with it was you hear about LLMs, these language models, and they basically were saying the difference between this SR Linux GPT and chat GPT is, you know, chat GPT uses the internet's history essentially up to 21 or 22,. Right yeah, whereas I guess theirs uses their own language model that's very network focused.

Eric Chou: 14:29

Right.

Danny Wade: 14:30

So you might be able to get some. It has additional context that maybe a chat GPT wouldn't that's at least the way I took it, so. But yeah, so it was pretty cool. It was a nice little demo for sure.

Eric Chou: 14:45

Yeah, yeah, I mean I'm sure there are going to be more use cases that comes out of it, but definitely you want to have some kind of intercept Right. There's a very famous example of 2 plus 2 on chat GPT proper and they would open AI and they would just fall on their face right, just because it's like, and also like the accuracy decreases over time because they're taking so much more data and that includes both good and junk so that initially they might be able to figure out, if enough text out there have 2 to the power of 2 to be 4, then that's great, then now went out. But if enough people who are putting in junk like 2 to the power of 2 equals to 8 or 16 or whatever, then eventually those data will win out right. There's no intervention. So this seems like Nokia is intercepting the request and they try to answer it themselves first or at least construct a prompt that get back into an answer. That's underneath them. That has a lot of chat GPT, but the key information is networking and Nokia.

Danny Wade: 16:03

Right, yeah, I'll be interested to see where it goes. They did mention multiple times it's experimental. It's basically a pet project up to this point. So don't ask your sales rep for just a demo environment. And it's not like they're recommending to push it to all your Nokia devices yet.

Eric Chou: 16:20

No no or rely on your change management Right.

Danny Wade: 16:29

Yes, and so the next I think the last vendor that I sponsored that I mentioned here is Opsmill. So Opsmill is a new company. It's founded by Damien Garris and Raphael probably Butcher Munir.

Eric Chou: 16:44

Munir, that's how we pronounce it too.

Danny Wade: 16:46

OK, yeah, so I've met Damien and have a history with Damien, so I saw him I think the first day I was there and he kind of showed me around Opsmill and his kind of thought with it. So I will probably lose you explaining the product, but basically what it is is that it allows you to. So when you think of a source of truth you think of a NetBox, a Nautobot those are kind of two big players and that's where you store all your kind of declared state or your intended state of the network, or connections or IP addresses and so forth. But under the hood that's just a relational database and what Opsmill is trying to do is allows you to. If there's any sort of reason that you have to change your data schema to the database or the underlying quote, unquote source of truth, then the data changes go with your code and your changes. So it's very heavily integrated with Git.

Danny Wade: 18:04

So I'll kind of spin it to. If you've ever contributed to a project on GitHub, you create a pull request, right, it has all your code changes in it and then it goes through some CI testing and so forth. But what this allows you to do, this product it allows you to, the database can actually branch just like you do in Git right. You create a branch, do your changes and then you create a pull request to merge your changes. This is basically a database that allows you to branch the database with your code changes and then also create a pull request and include those database changes as well, if that makes sense.

Eric Chou: 18:48

Yeah, I actually just talked to Damian prior to jumping out like an hour ago and just specifically focused on Opsmill. It is very cool.

Danny Wade: 18:58

Perfect. Well then, what I would say is if you ever do a recording with him, go to that recording and Damian can explain it way better.

Eric Chou: 19:09

I don't think I could edit that fast enough Bigger Up, but yeah, I recorded a session with him and he talked about just as what you were saying, danny, on the challenges that he see, and, of course, eddie's here on the chat and he created this tool, naa, which we released those episodes a couple of weeks ago and he wrote about it on the kid on the back end.

Eric Chou: 19:38

So I think there's definitely pros and cons on the two different approaches.

Eric Chou: 19:42

And something that he probably mentioned in the conversation but you haven't written it down here, was he uses graph database as opposed to a relational database. Because of that challenge you mentioned, you have to predetermine the relationships between entities and it's kind of hard to do that ahead of time if the relationship is dynamic, if it's super easy. If you know, you have two data centers in this region and each data center has 500 devices and each device are this type of device and that type of device. But it's hard to capture the metadata on, say, a network design through several iterations where you have some video, some people put in their reviews and some people are swapping out this and then coming up with a decision on 64 uplinks versus 32, or something like that you could capture that 32 end result, but you can't capture the reasoning behind it, essentially the discussion up to that point. So then the next person comes in and say, oh, 32, that's great, but they don't know the why behind, or the month-long discussion on why that is the number they picked.

Danny Wade: 20:58

That is a better explanation than what I just gave. So, yes, no, no, no, absolutely. No seriously, that was yeah. I definitely missed that in the blog post. It's yeah. If you've never heard of graph databases, I recommend taking a look and googling around and seeing the power of graph databases. It's very, very interesting.

Eric Chou: 21:21

Yeah, I've heard of GraphQL but I never heard of graph database. I didn't know it was already kind of productizing a lot of open source project. I've heard of NoSQL and I've used a lot of them. I use SQL, but it seems like a very interesting movement and something that Damian mentioned. I'm sorry to cut you off, but something that mentioned really strongly about was every once in a while, as network engineer, should peek out from our, look up from our, without burying ourselves in the screen, and look up and look at other industries. Other people have solved this before and we should take the good parts out from their struggle or their wisdom and apply to network engineering.

Danny Wade: 22:11

Yeah, that's exactly that, I would say, is the theme of network automation, and what will get us to the next level is exactly what you just said, is look across different industries, specifically software development and they've been doing what we're experimenting with for years. For example, ci CD right, it's a very crazy acronym for some and it's very buzzwordy in the network industry, but you look across software development. That's a tool in their toolbox. So, yeah, we definitely need to look across other industries.

Eric Chou: 22:45

Yeah, but the challenge then I posed to him in the section like my little plug is it's so hard like we're already overwhelmed with our daily jobs and family and trying to, it's hard enough to keep above water, let alone like go into a whole new, different vertical where we know nothing about or very little about, and you feel like you're wasting time because you're not the level that other people at least I mean going to like a data science conference, right Like I feel like I'm not being productive because I'm so far away from understanding what they're talking about. What's your thought on that?

Danny Wade: 23:25

Yeah, I mean that's yeah, it's the nerves whenever, the nerves you get whenever you're learning something new, but you don't feel like it's applicable to you if that makes sense. Yeah, I can see where you're coming from, but I think there also needs to be kind of a paradigm shift in networking, right? I think I forget who mentioned it. Actually, it was kind of a repeating phrase and you've probably heard of it as well. You have to start treating the network and the network devices as cattle instead of your pets.

Eric Chou: 24:02

Sure, I think it was the organizer who kept on. Oh, no, no, no, oh, it's no. I mean, it's the Alex, alex Henthrone Iwan, I think that's that. He first came out with it, and I only know because I'm scheduled to record that session with him this afternoon.

Danny Wade: 24:22

Oh, there you go. So perfect example, right, and that's kind of that like. Obviously you could take that a couple of different ways, but the way I look at it is you kind of just have to treat the network at the end of the day, as just data, right? So what we do with the data, how we collect it, how we push configuration, how we get that full that feedback loop, there's many ways to do it, but we need to start looking at the network as data and not as basically artwork, if that makes sense.

Eric Chou: 24:55

Yeah, like you back into that cattle versus pet. Right, you babysit your pets. When you go on vacation you check into a hotel, but in cattle it's just like any other uniform transactional thing that you do on a daily basis, Right.

Danny Wade: 25:14

Yeah, I mean it's, it's um, yeah, the we just have to look at it differently to where it's. You know, you may and I haven't really run into this professionally, but you know you hear stories about people that may. It might be a senior architect or someone that's just been there a while and it's like oh hey, I configured BGP for the network, or you know, I did that or I did this and it's like, it's cool, like I'm glad that you had the thought and the design and like it is, you know, a really good design. But the implementation steps shouldn't be anything to talk or kind of flex about, if that makes sense, right? Because it doesn't.

Danny Wade: 25:50

It shouldn't matter how the device gets configured or the little, the tiny little configuration options that you went through and hand jammed it's. You know it's good to have a good intended design. I don't. I think you do need to implement those merit knobs when necessary, but I don't think we need to worry about how the device is configured and then how we validate the network. I think that's something that we should kind of have an open approach to and understand that. Okay, we have a bunch of data from the network. Let's figure out what the network's doing, the intended state and then how we need to get there. And that's where a lot of programming and Ansible and things that you know we're we're talking about constantly network automation. These are tools that help enable that.

Eric Chou: 26:34

Yeah, I think that goes also goes back to what you were talking about in your conversation with Damien, right, because those, those design process were not captured, so those often lived inside of these. You know, people who came up with it in the in the first place uh mind or like maybe in the wiki or whatnot that nobody reads, so so they. That becomes a unique snowflake process that others, even for a motivated individual, cannot understand and that becomes like this treasure that lives inside. But if you can capture, uh, somehow you know in that, you know reviews and everybody who had inputs, at least as a newcomer, right, I just joined the company yesterday and I'm able to read all the documentation and as a, as a smart, motivated individual, I could contribute to that, versus I have to go back to that person and buy him a beer or whatever he and take that out, right?

Danny Wade: 27:33

Yeah, that's exactly exactly Right. And funny enough that that kind of puts us into this next section people and processes kind of leads right into it, right? So, yeah, so, people and processes, you know, I've read about a couple of the speakers there. Um, you know, we'll go through each of the individual presentations. Uh, they'll be. I believe they said, um that that the leaders of NAF are going to post the um YouTube record, or recordings, to YouTube. Um, that way everyone can take a look at all the different speakers and whatnot.

Danny Wade: 28:08

Um, obviously, the ones that the one that highlighted in my eyes was Jeremy Shulman. Okay, he had a really good presentation about um, basically a design driven approach, um to network assurance and so that what that means is not just taking in network design and configuring devices, but also, based on the design, dynamically creating tests and validation points to make sure the network looks and appears as it should. And we're not talking just like hey, is the CPM memory look good? Does you know the control the? You know the control plane look good? It's everything down to the cabling. Are we, are we seeing? You know transceiver? The light levels look good? And it was very in depth. Um, and I mentioned, I put a link to his um. He had a presentation for Nanog um that kind of talked about.

Danny Wade: 29:05

I think it was more theory, um, but the one the presentation he had for NAF was very demo driven, so it was. It was really cool to see his tools and what he's built for for the MLB.

Eric Chou: 29:20

Sorry, I mean, I didn't mean to get that, but yeah, so the link is uh in the um, in the in Danny's uh write up Right.

Danny Wade: 29:29

Yep, so I mean, um, so his was good. Another one that I didn't I don't believe I included, but uh, I would recommend checking out. As well as Jason Davis. He works at Cisco. Okay, um he has?

Eric Chou: 29:43

is it the one who launches the Cisco live um uh like units and then network inside of a closed loop system?

Danny Wade: 29:52

Yep, yeah, so he, he has a? Um pretty big focus on observability, um telemetry. He's built some really cool dashboards, um, and, yeah, his he's probably the most famous for the uh knock dashboards, the Cisco live knock dashboards that he's created um and kind of presented at Cisco live on. So his, his uh presentation was really cool and I would recommend that you know that one as well. Nice, but yeah, the presentations, uh, you know, I definitely don't want to take any steam away from the people that presented. Just go check them out whenever the recordings are released and, um, I think you'll enjoy. There really wasn't one that I would say, oh, like you know, it's just kind of a repeat or something you always hear.

Danny Wade: 30:36

Everyone had either, you know, a different take on how they're deploying automation in an enterprise. So, um, I believe I mentioned um Kion, yeah, from the times, uh, he deploys the way they use Ansible and deploy network automation. You know his company was different than um. I don't know if I wrote he works for 1111 systems. I think it was Garrett Noak, I think it was Garrett. Um, he for 1111 systems. He explained how they, in a year or year and a half, I believe you know, tried doing a transformation or completed a transformation to automation. Um, but the theme, the theme I thought that was interesting across the board was you know they said they've implemented network automation, Um, but they're not locked out of the CLI, right.

Danny Wade: 31:31

They kind of I always say, they kind of do it like a reality check, right. So they they said, hey, you know we've adopted network automation, but you know, in reality, yes, you still have to SSH the device to do some, you know, to kind of put on your network engineering hat, right, Cause there's sometimes you need to be in the device and troubleshooting and whatnot. So it was kind of cool to see that balance between.

Eric Chou: 31:54

Yeah, I mean, why would you ever lock yourself out of I mean at least console, right, like just ask. You know the people who experienced the outage at Facebook, where like out of vision is prevailing and the predominant way of doing things. But you know the guy who was the poor guy who was in charge of fixing it when it got all old and you know, like he still needs to go check and he's batch expired right During COVID. So why would you ever lock yourself out of that?

Danny Wade: 32:23

Right, no, that's exactly. I think that was actually a joke that I heard in the hallway was after one of the presentations was oh well, yeah, you don't want to turn to Facebook, but yeah, I mean yeah, you don't want to, which we'll put that into like a term, right, you don't want to Facebook it.

Danny Wade: 32:38

Yeah, right, yeah, I mean, that's a good point. You just you want to. At least you have want to have the ability right. And the whole purpose is that we're saying this for people that might not really understand. Like, why would you ever want to lock yourself out, Like for real? Yeah, A lot of times you might be. You might hear in the automation like, oh well, when you implement automation, we can't have any sort of config drift or any like manual changes, because that messes with the automation, right, Right, and so that's, that's what, at least what I've heard. Right, I'm not an advocate of that, it's just if you've ever heard that it's. It comes from that vein of oh well, if you know, if you're still allowing your engineers to manually log into the device, then what's the point of automation If they're going to mess it up and divert from the standard instead of fourth, but yeah, yeah, I think the balance that I've seen before was the fact that your your manual change will get overwritten in like two days.

Eric Chou: 33:37

So if you're like fixing a bug and you you know you you need to hand change some stuff just to fix a bug, you better like make it permanent or semi-permanent with the manual Within the next 48 hours. Otherwise the automation tool is going to come in and say this is not what it's supposed to be and you know reverse or change.

Danny Wade: 33:59

Yeah, and I'll kind of leave with the point. I actually heard or saw that Dartmouth College they they've had webinars with NetBox to explain kind of their strategy with automation, how they've adopted it. I'm supposed to. You know it's NetBox focus but what they do is they generate configs and I believe they said every single night they do a config replace on all their devices with the generated config sourced from NetBox using the config context and I thought that was pretty insane. That's hardcore. You know is hardcore. Yeah, no, not insane, is like a dumb idea or crazy.

Eric Chou: 34:44

Sure Sure, I think it is hardcore Like that is.

Danny Wade: 34:48

I mean, every single night I'd be worried like okay, it's almost midnight, the whole network is going to go through a config replace here. But if you're interested in that, I think there's probably a few different YouTube videos out now. Definitely recommend checking it out. If you look up like Dartmouth College NetBox, I'm sure you'll find one one of the videos and it's very interesting to see the trust and automation and you know the configuration reliance there.

Eric Chou: 35:16

Yeah, and sometimes they just do it for other reasons, right? So I think I you know my old coworker, justin Peech. He wrote from Amazon, I mean, he wrote about there was a time in place where there's significant scaling issues with low balancers. I won't mention the vendor, but he wrote it in the blog post that there are times where they just daily reboot the box, right, and so in terms of automation, manual changes, they just you know they're, they're overwritten and you know, whenever they reboot it, you know they're not going to save the running config, they're just going to reboot it and then, once they found out that it doesn't, the impact is not as severe as one might inspect then every single time they make a change, they just reboot it, right, it's not even you know, they write it into the running config and save it. They just like write it, run it and then reboot it, just to to avoid any kind of memory leak or anything like that. Yeah, that was yeah anyways. Yeah, so that's, that's just a follow up on what you said.

Eric Chou: 36:33

And if I look at the chat right now and so Eddie mentioned that you know the network side of IT is five and not 10 years behind the server side, and you know. And then the network vendors haven't made their product compatible with network automation. It is true, I would definitely think that not definitely think, but I would agree that it's less commoditized. So the server, you know, is really a commodity, right, like I could buy a Dell or a HP, and it doesn't impact my ability to load Linux onto either one or the flavor of Linux, but, you know, try to try to, you know, load Junos onto Cisco, right, like good luck with that. And yeah so Kion Thana's presentation in OtterCon, yeah, so me too. I mean, I would definitely look forward to what he's done. And it sounded like he may not be the only guy, but they sounded like he is kind of the only guy who's doing it at New York Times, right?

Danny Wade: 37:35

So yeah, I believe he's, he's definitely leading the charge. I was able to talk to him a little bit the last night of the conference and he said that you know there's definitely some people that you know most of the teams on board. There's a couple of people, stragglers, you know, that would help, you know might need some additional help, and I think that's across the board right when, if you're trying to be, if you're an advocate, evangelist for network automation, you can get a couple of people convinced. And then it's a process, and I mean this part of the blog that you know you have on screen now the people and processes. It's. It takes a while and that was a theme that kind of I came away from the conference as well. Is everyone that presented that were that like, like Kion he, they explained that it's not a overnight thing, it's not a week project, it's not, you know, a hackathon project. It's not a month. This is a multi year project.

Eric Chou: 38:38

This is a it's either right here automation is a journey, not a race.

Danny Wade: 38:42

There you go. Yeah, so that's. Yeah, those are the my take the wrap up to the blog and yeah, it's exactly what. That's exactly the theme I took away. Everyone says it, but whenever you actually hear people's journeys and their explanation of, yeah, you know, like, for example, k-on, he told me, I believe he said they're in year four of this adoption. So you know, four years and there's a lot that goes on in four years and it's not just you might, over that long time period, you might not even be trying to convince the same network team that you started with right.

Eric Chou: 39:18

Because of people. Four years, you're like graduated from college or high school.

Danny Wade: 39:22

Right. But you may be, people might retire, people may leave. So like it is a journey and it does take you know. Another point that I make here is it's a large process change and it requires buy in, and not just from the people that write the check for the tooling, but your people, managers, the rest of your team. It really requires kind of everyone on board, because as soon as you have you know a few people that are against it not people that just haven't picked it up, but people that might be against it then it starts to crumble and it crumbles pretty quickly. So it's that's kind of the. Those were, at least you know, a couple of the themes that I took away.

Eric Chou: 40:07

Yeah, and I also want to thank you, danny, for writing this right, because it's a. You didn't have to write it and you didn't have to put as much thought that you would have into this for those of us who couldn't make it and, of course, you know, sharing your, your take in person. So I really want to say thank you for for doing this. This is well, well done.

Danny Wade: 40:30

Absolutely yeah, I mean, you know, I'm not near my recording station.

Eric Chou: 40:35

But I would give you like a plot sound, but I don't have that right now.

Danny Wade: 40:39

No, I. But yeah, the reason I do is, you know, you know me, I'm an advocate for network automation and I I try to. I love helping people and trying to help explain things to people and you know, this is one thing that if you weren't there, I just felt that you at least need to get the messaging. And there's a lot of people that wrote blogs about kind of recapping, like some people did, like a blog on day one and blog on day two that are more detailed and have different takes, like you know. One thing that I'll just say like my blog definitely doesn't cover, doesn't cover all the different vendors, doesn't cover and there were plenty right, but I didn't want to turn this thing to a book. I just picked, you know, just handpicked a couple, but there are plenty of sponsors there that I would go into detail for and other people that I've met. So it was a very cool experience.

Eric Chou: 41:32

No, I would say that you offer the most value from your perspective, right, if I wanted to hear from the vendors perspective. Actually, just get on their website and, you know, watch videos later or whatnot. What is really valuable is from your perspective, your background, and looking through your lens then it puts me into your mindset and so on. So, no, definitely, you know, pick out the ones that you see valuable, right, that in itself is valuable.

Danny Wade: 42:00

Yeah, absolutely.

Eric Chou: 42:02

Cool, all right, cool. So we're going to open up, you know, maybe like the Q&A session If you're free to just ask the question yourself. But I would go ahead and start with the first one, that Keith. So Keith was there as well and he works for Juniper. And the first question is where are the vendors failing in your opinion? Right, that one, I'm at it, you know, obviously he's too close to Juniper to see out the issues. So what do you think the case is? And I'll add my piece from hearing from Damien earlier on. But go ahead.

Danny Wade: 42:39

Oh, man, put me on the spot here. Yeah, I'm just kidding.

Eric Chou: 42:43

I love it. Feel free to take out that soapbox.

Danny Wade: 42:47

Oh yeah, no, I like answering these sort of questions because they're the tough ones people like to run around, so where the vendors might be failing. So I think the one piece and this was a discussion point that I believe there's actually going to be an action taken with it, as well as just the different jargon used within network automation, okay, kind of the different words that are used so you know people talk about. I'll use configuration as an example, right, so in Ansible you have a concept of playbooks. Sure, outside of automation and just in regular network engineering, and this, this may just be company lingo, but a lot of companies you know you work for or work with playbooks are considered like your implementation plan. Sure, like your mom, like your mom, right?

Speaker 3: 43:46

Yeah so.

Danny Wade: 43:46

I've heard it yet, my playbook so. But some people take that and try to move over to network automation.

Danny Wade: 43:55

They hear Ansible playbook and they're like, oh, like, what's that about? That's like one very simple example. Another example would be templates, right, sure People? What does that mean? Well, an automation Gingert to templates. In network engineering we have config templates. Config templates and network on or network engineering are literally just text files with a couple words, you know, a couple lines of configuration on it. Gingert to templates, something completely separate and that's very specific to a Python library. Right, it can.

Danny Wade: 44:26

It's just that there's when you're talking about network automation and network engineering together, people kind of can mix lingo. Like I mean, you can talk about hey, I read a playbook for this. Someone that doesn't have too much automation experience may say, yeah, I write playbooks all the time for our network changes, and I might hear you and say, oh, wow, that's like, that's awesome, like glad you guys are adopting network automation. Well, no, no, no, I'm, you know, I write down. We have a change template and I write down these, all these commands and anyways, that's very. I think that's a super simple example and probably not the greatest, but I hope you kind of get the understanding of like we just kind of have to outline all the jargon and all of the different wording we use within automation and then how that kind of works with network engineering specifically.

Danny Wade: 45:17

So that's one piece, I think, because a lot of times you get caught up in just word salad with some of the vendor products. But I mean, besides that, I think it's really tough to be a vendor and then also try to hold that competitive advantage with automation, because a lot of the tooling around automation is open source, right, so you can take and build on those open source products. I mean, a good example would be you know, jeremy stretch built net box and now net box Jeremy is part of a company called net box labs that kind of builds on that and provides product ties marketing behind it. So there's ways to kind of market these, the automation tooling. But I think it's really just the takeaway from it is take a step back, try to understand what you're trying to solve, what you're trying to help, and not try to upsell some other product with it, if that makes sense, right.

Danny Wade: 46:22

So don't don't say, hey, we're selling this tooling but it only works with our latest switching model or latest router model or something, and it's like okay, well, does that mean I have to buy this too? Yeah, you do. You know what I mean. Like it, just keep it open as possible. And so that way, because whenever you go into buying a product or helping a company buy a product, one of the first things you look at is what's the vendor lock in? Yeah, many times no one's going to say yeah, give me the one with the most vendor lock in.

Eric Chou: 46:55

Yeah, so it's almost like the cell phone model, right, like the cell phones, free. But then hey, you know, if you want some plans, go with it then. And, by the way, it's going to be like 36 months, three years, locked in, right. So you know, like the, the software is open source, but you do have to have our latest and greatest hardware to go along with that. Nice, you know, automation.

Danny Wade: 47:18

Right, and I will say this there are times it makes sense because maybe there's a feature or something that's just not present in older hardware.

Eric Chou: 47:27

Yeah.

Danny Wade: 47:29

Right. So I know it's a balance, but I think this day and age I would definitely try to stay as open as possible without forcing any other upsells.

Eric Chou: 47:41

And then that this is not unique to networking, right, so they're. You know software there's, you know kind of a what is the like end of life, end of engineering, end of like you know patch, right, so it's only usually like in the three year span. So you can, you know continuously, you know there's something that was 10 years ago, but at the same time you know you have to have some sort of public release and give people time to react and reasonable time to, you know kind of get caught up with it, right, so it's okay if your automation tool doesn't support, you know, ios without, but you know you have to at least not having that refresh rate every six months.

Danny Wade: 48:23

Right, and I guess one last takeaway and we can, we can move on if you want is please, please, please, don't put any of your product API is behind a registration or paywall. I only asked that because, at least from my perspective, you know if we're looking or considering tooling, whether you know for client, you want to be able to investigate the integration points of the product and ways that you can integrate with it, because obviously there's companies that you know, for example. It's them, they may have service now and that's staying right. That's not going to be something that changes within the next five years or so. So you're going to look for products that can potentially integrate with that.

Danny Wade: 49:10

Obviously, service now is a big thing that a lot of products do have out of the box integrations with. But let's say you do have to look at the bigger picture and say, well, I want to integrate with that, I want to integrate with IPAM, and so you start looking at the API is available to see where there's integration points can lie. So that's that's kind of where, where I'll leave it with nice.

Eric Chou: 49:35

Yeah, I mean, and sorry about that sound, because I was just clicking on different links that Keith send before and I'll include these in our show notes for sure. And first was the net box labs with the Dartmouth College that that Danny mentioned before, as well as a, you know, achieving our automation with the YouTube. It's the same one. So the second question that Keith has was is there any value if we begin to offer some kind of suggested practice for automation, maybe multiple ways to automate the same tasks, the same task?

Danny Wade: 50:16

Sorry, I'm going to re-read the question here.

Eric Chou: 50:18

Oh no, no worries, I think what he meant was you know, if there's a value, if we have like kind of a cookbook or a collective best practice place to do, that is that. Is that what you mean, keith Almost? You are muted.

Speaker 3: 50:35

I don't want to call it best practice, I just want to call it suggestions, like to help people who are just getting started and have a few ways to look at how to automate something and then not be overly opinionated about saying do it this way, but say you could do it one of these three ways or one of these four ways, one with Ansible, one directly with Python, maybe using different modules. So the idea was should we start that? Is there any benefit to doing that? Because something that came up at AutoCon in my conversations, where people were kind of operating in a bubble they were lone wolves, getting it done but not knowing if it's a better way to do it, If there are things they're not considering. So my thought was should we start trying to pull that together?

Eric Chou: 51:20

Yeah, I mean. Well, besides, I mean, since I have you here, right, we'll answer that question just a little. Not well, I will let Danny answer that question just a little bit, but since you have you here, what's your overall take on the AutoCon event? Do you have anything to add to that?

Speaker 3: 51:37

My take on it was it was encouraging to see that many people that interested in network automation to make that trip, you know, even if it's just time away from their normal jobs, to engage and be there for those conversations. That's really encouraging. Some of the environments that I've worked in you'll find that you're the only person that's excited about automation and what it can do. But in that environment you were one of many people excited about what network automation can do, so that was encouraging to me. Anyway, I was surprised to hear how strongly the opinions for vendors being the big source of the issue.

Eric Chou: 52:18

So one thing that I was honestly but again, did you like have some like nervous chuckle and be like, yeah, I work for Junferto.

Speaker 3: 52:27

You know, it was plastered over my badge. I mean my badge. Even though I wasn't there on the company's dime, it still showed me as a sponsor. So I wondered if that's why I heard it so much.

Eric Chou: 52:39

People are coming out to complain.

Speaker 3: 52:41

Yeah, I mean I don't know, but I even ping ping you. I sent you a message and I sent a few messages on LinkedIn just to ask, like you know, what are the vendors in general doing wrong? And I do want to know what Juniper's doing wrong, along with the other vendors, so I can have a better perspective Because, again, I didn't know that that there was a huge problem around that. So the event was great, though it was really encouraging that I appreciate the time that you took for the blog and going over things today, so I was just encouraged in general.

Eric Chou: 53:11

Yeah, I think that was the the took away. One of the took away from Damien as well. He was telling me about a the previous one that was sponsored by Network to Code, and this was like pre COVID, so it must be 2019 or before, where they have to latch on to another conference which was interopped, or I think so it was like interopped and then like a day after or maybe like a sub session with the interop and there was like 80 people who showed up and this time I think he mentioned there was 300 plus. So, yeah, that was one of the the took aways.

Eric Chou: 53:44

Oh, and something that I wanted to mention from your previous question that where the vendors are failing, and I think part of and correct me if I'm wrong, you guys were both there, but I think Jeremy Shulman was actually on the stage, you know, expressing his opinion on many of these stuff, that he shouldn't be doing it by himself. Right, like there should be an open source project. Like more people should participate in it. Right, like they should participate, they should open source it, and then they should. We should all collaborate as opposed to one or two heroes. You know Kirk Byer being one and you know NetBox being another and all of that. Like they just they shouldn't be showdering all the weights. Yeah, what do you think about that, danny?

Danny Wade: 54:32

Yeah, yeah, he did mention that towards, towards the end, I mean that was. He mentioned it in the context of I don't know what the question was, but it was related to, like you know well what do you? Think like not, it wasn't the build versus bi-conversation, but it was kind of like that where it was like well, you know, what would you prefer? Do you like building your own tools or what do you? And I've you know?

Eric Chou: 54:53

he said to the effect of yeah, I mean, I like to be able to buy a tool from a vendor that kind of checks the box, and obviously that that box checking is different for every organization and every person, right, yeah, so I think previously people were the way that you know some vendor have done it before was, and you know, like the vendor you may have mentioned in your blog, may or may not mention your blog too that you know they, they take the biggest you know box checker and say if I check box for all, like if I check all the boxes for this guy, then it should be good enough for like 99% or 90% of the other people who wants to automation, right? So yeah, you know, I see. I see you know arguments or pros and cons from both sides.

Danny Wade: 55:46

Yeah, and just to add to that as well. So that was. I think these are two different questions that were asked whenever it was like the final panel, I believe, is when we started talking about this. The last other question, which kind of relates back to what you're saying, is he. He basically was saying you know an open source? Because a lot of people were saying, hey, yeah, like you know, open source is great, like you know, keep everything open source and whatnot, and it is. But what he was saying was there's a lot more consumers than producers and open source and specifically in network automation, for sure.

Danny Wade: 56:17

Just saying hey, like it was kind of a call to action, to say, if you're interested in open source tooling, you like using the available open source tools, look at the list of issues, right, they're public. Look at the issues and maybe try to do something small update documentation, fix a small bug, or you know you don't have to tackle, like, add a new feature to net box or anything like that, right? So I just encourage people I'll say this much because I just started getting more involved with contributing to open source within the past I would say year and a half Sure, just if you run into a problem and you know you've tried a couple of different ways to solve that problem, using a framework or a library and you're like this is weird and I feel like other people would you know it would work out for other people and help others Open up an issue. Right, there are so many people that are looking to contribute, or at least some of the maintainers that are looking for just that feedback loop. Now I would say, if you open up an issue, I would say at least have a thought of whether you want to contribute or not. You know, try to fix the problem.

Danny Wade: 57:28

I guess If it's a really big issue, obviously you know I would still open up an issue. But but yeah, like, just be just cognizant that these are, you know these are public projects and kind of come into it knowing, hey, I want to make this project better, I use it all the time and because at the end of the day, we're, you know we're all in this together and so we really any way we can help each other out.

Eric Chou: 57:55

Yeah, I mean I would Roger that. I mean I contributed open source, I sponsor open source projects and you know, if everybody give Kirk a dime, if everybody who uses Kirk's work, a Kirk buyers work, give him a dime, he probably doesn't have to work and he got concentrated exclusively on that, miko or Neypom. So but but having said that, I want to say our denominator is smaller. You know, compared to people who use Linux or people who uses Ansible right, like the people who does out in the round nation or network engineering general is smaller. So they're not denominator smaller to begin with.

Eric Chou: 58:32

And also the this issue is not unique.

Eric Chou: 58:36

To narrow down to mention like meaning open source, maintainability, sustainability.

Eric Chou: 58:42

There are, there are projects like request that's being used by millions of people and by billion dollar fortune 50 companies and they contribute nothing to the project, right, so it's not unique.

Eric Chou: 58:55

And something that Python and I mentioned this in my recording with Damian as well is that the Python software foundation have came up and say, ok, I'm going to be the 503 nonprofit that covers all of these. And you guys just coming under my wings, like your you know your Python meetup in Seattle, your pie data meetup in San Francisco all just come under my wing and so you know, microsoft could write me like a $10,000 check and I distribute that to you so you don't have to fill out the paperwork, have a board of directors and having board meetings every quarter to do that, and I think that's one way to have open source maintainability right Is to have something like a PSF, the Python software foundation, come in and, overlooking all these other smaller projects, and you know they've been pretty successful right. Like you know, flask was a project that was near, you know kind of not maintained for a couple of years and then they took it down and now they're, you know, thriving and alive and all that.

Danny Wade: 1:00:00

Yeah, I mean that would be great if, yeah, kind of having that. Yeah, the funding and management, yeah, and, like you said, it's not, it's not a problem. Just in network automation I mean contribute the consumer versus producer ratio and open source is ridiculous across pretty much every open source project in every domain. So but since we're already a smaller group, you know, like you mentioned, like a, it would be very cool to see just if you, you know network automation engineers or infrastructure automation engineers, whatever the title is, in the you know 510 years, if they come in and one of the first things they learn is get and how to contribute to open source, because that's what the person before them did and has taught them. I think that'd be awesome.

Eric Chou: 1:00:47

Yeah, I agree, and this, this is a. This is a bigger topic, right, and this is, it's just the fact that we're talking about it is an improvement in itself, other than just like, how do I write the script or how do I, how do I write this playbook, as you mentioned, right? Like and very tactical stuff, right? Just the fact that we're elevating into an industry-wide thing is a giant step forward. And if I just look at the rest of the chat, I know we're almost out of time, but yeah, so Autocon one is going to they're already talking about it. It's going to be in Europe in January and I believe they have auto Autocon one schedule or on the horizon in fall of 2024.

Eric Chou: 1:01:31

So, yeah, it's happening pretty quickly. And from Jose says, what I hear, what I hear a lot is what should I automate? Should I keep everything under automation? I think that's probably a bigger, bigger project and I agree, like that's a topic that come up a lot. But, as Dan was talking about, on the people and process, especially like K-ons talk and all that is that it's a multi-year process and probably don't want to take that SSH or console access away and you don't want to quote, unquote, facebook it.

Danny Wade: 1:02:07

And just to answer, jose, that's actually one of my takes that I did mention. So I had three takeaways. So the one I didn't mention was start with low hanging fruit and build on those easy wins. That was another theme that I took away from the conference. So I mean, I think I started with talking about doing software upgrades. That's kind of, when you break it down can be, I would say, more intermediate problem. When we're talking about difficulty, right, because there's a lot of validation checks and you're rebooting a device.

Danny Wade: 1:02:38

But think of the thing I always preach is think of read only operations, right. So the one thing I always tell people is look at test automation, look at network testing, and what that means is, yes, you can look at configuration. You can do show run pipe, include whatever, make sure that that configuration is present. But what I really mean is doing show up, you know, show commands to look at operational status of your routing protocol or your layer two spanning tree, or looking at those sort of tables, being able to parse those tables and look for that specific value that you're expecting. Doing that in an automated fashion is a very low risk task but you get a bunch of value out of it because you really learn how your networks operating.

Eric Chou: 1:03:25

Well said, well said, danny. I mean, I know I know we could probably talk on this subject for like another hour or two, but I want to respect your time and as well as people who joined today's sessions time. So where where can people follow you find your work besides this excellent devnet dancom blog.

Danny Wade: 1:03:44

Well, thank you for the promotion there, so you can find me on I guess you speak all Twitter now called X at DevNet Dan same name my blog and I'm also on LinkedIn. Not on LinkedIn is DevNet Dan, just my name, daniel Wade. So, yeah, but I've been. I've been promoting my content blog on LinkedIn more and so, yeah, you can follow, follow me, connect with me on there.

Eric Chou: 1:04:17

Awesome and thank you everybody for joining. You know let's keep in touch. My little plug is join the network automation community on members that they were got a mission, that community, and you'll find everybody who have contributed to today's chat and, of course, dan on the on the community and thanks everybody. I'll go ahead and hang up and everybody have a good one. Okay, our mission on.