S1, E6 – Navigating the Cyber Storm: Lessons Learned from DMEA

[00:00:00] Speaker A: One, two, three. Come on. [00:00:02] Speaker B: In the grand stadium of the digital landscape, where teams of innovators and hackers face off daily, there stands a commentary box like no other picture the buz of a pregame show, the strategy analysis of the halftime break, and the deep dives of a postgame discussion all rolled into one. Welcome to off the Wire, a play by play on cyber issues, your sports desk for the big league of cybersecurity. Just like in sports, in the cyber world, understanding the play is key to staying ahead of the game. And that's exactly what we offer. A podcast that brings the strategy room to you, helping you to understand the moves, the players, and the landscape like a true analyst. Off the Wire, a play by play on cyber issues. Welcome back to off the Wire, a play by play on cyber issues. John Watkins here. We've got quite the exciting episode for you, don't we, Anthony? [00:01:02] Speaker C: Yes, we do. [00:01:03] Speaker B: So I think we've been pretty busy the last few weeks, and I know I have and been doing some traveling and that kind of thing, and looks like our numbers just keep going up. [00:01:13] Speaker C: Yeah, they keep going up. And I think we have a total of 75 followers between Apple podcasts and Spotify. [00:01:20] Speaker B: Yeah, that's great. Just under 100. So it's getting there. Now. This is our 6th episode. This one will prove to be quite the exciting episode, and I think everybody will want to take a listen to this one. Anthony, what are we talking about today? [00:01:32] Speaker C: This one is navigating the cyberstorm lessons learned from DMEA. [00:01:37] Speaker B: Now, who is DMEA? [00:01:39] Speaker C: Delta Montrose Electric association. They are a co op in Colorado. They had a cyber incident in 2021. And one thing I'll point out is we have used them as an example because, well, one, they're a co op, and two, they're a co op really close to the same size that Fort county is that I work for. We're at 35,000 meters and they're at 36,000 meters. [00:02:01] Speaker B: And they also have broadband service, too. [00:02:04] Speaker C: They do. [00:02:04] Speaker B: So, yeah, I was just out there and doing some work with John Watkins Consulting, which is, of course, our sponsor, which we probably ought to roll that real quick. Let's do that and get that out of the way. And let's take a quick listen to our sponsor today. Today's episode is sponsored by John Watkins Consulting, the cybersecurity Expert Electric Cooperatives Trust. Are you an electric Coop struggling with limited resources and the daunting task of safeguarding critical infrastructure? John Watkins consulting specializes in easing that struggle with over 14 years of experience, John offers tailored solutions to fight your unique cybersecurity challenges. Don't let the difficulty of board engagement and cultural challenges hold you back. John Watkins Consulting will guide you through these challenges, turning your pain into progress. [00:03:00] Speaker A: Yes. [00:03:00] Speaker B: Ready to fortify your coop's digital defenses? Call 937622 Eightyn or visit John watkinsconsultant.com. John Watkins Consulting turning your cybersecurity struggles into strengths okay, well, thank you for that sponsorship message. Mr. JOHN WATKINS Consulting we appreciate that. And if you have any cybersecurity things that you need taken care of, get a hold of John. Myself, actually, and I'll be glad to work with you, but let's get right into it. This is kind of longer than normal, but just to kind of queue this up, like I said, I was out there doing, I was on an engagement at Delta Montrose. I know Bob. He's a great guy, and he graciously said he would come on the podcast and tell all about the cyber incident. Now, one thing I want to say about this is just like Anthony said earlier, everybody's been using these guys as an example of maybe not what to do and some lessons learned and all that kind of stuff, but you got to give a lot of credit to Bob and his team because to be honest, there's a lot that goes into this, and it takes a lot of guts, in my opinion, for him to come and talk about it. [00:04:10] Speaker D: Right? [00:04:10] Speaker C: Yeah, I think for a lot of co ops, just whether you're a co op or any entity, just burying your heads in the sand afterwards is what people are naturally going to want to do. And I'll say this, there have been a couple of co ops that I know that have had cyber incidents and there is no whiff of news about. So, you know, folks may think that this isn't happening as frequently as it is. And anyways, I really appreciated them sharing a story time and time again. [00:04:37] Speaker D: Yeah. [00:04:38] Speaker B: And, you know, maybe a warning or whatever you want to say, but this is a very powerful interview. And again, a lot of kudos and my hats off to Bob Farmer and his team and all the work that they know, bless their, was they really took it on the chin and it was real stand up of them to come out and just kind of bear their soul and talk about how this incident impacted. You know, the best part about it, Anthony, is the fact that there's a lot of resilience. You know what I mean? Like, you can tell there's a lot of resilience. And when I was talking to, like they're just going, you know, that's the know, this cyber thing's real. It's going to hit us. But, you know, the fact of the matter is, so are storms, right. So are these other kind of outages and things like that. But as co ops, we've just got to just buckle down and just keep on because we're critical infrastructure. [00:05:24] Speaker D: Right? [00:05:24] Speaker A: Yeah. [00:05:24] Speaker C: And another thing I'll say is this just showcases cooperation among know. All of us have learned, and I'm sure several of you have heard Delta Montro speak or Bob speak. You know, of course I'm biased, but I will say there were some things that I picked up on this conversation that I had not heard in the past. I think Bob gets better as he discusses what happened, and I definitely think this is the best conversation yet. [00:05:49] Speaker B: All right, well, without further ado, we'll go on mute, and then we'll just go ahead and listen to what Bob has to say. [00:05:58] Speaker D: So, yeah, I'm here with Bob Farmer from Delta Montrose Electric association in Montrose, Colorado. Make sure I say that correctly. And, Bob, thanks for allowing us to interview you and talk about something that's not very comfortable to talk about. [00:06:15] Speaker A: Absolutely. Thanks for having me on your podcast. [00:06:18] Speaker D: So, just to kind of give a small overview, Bob's cooperative back in 2021 experienced a pretty severe cyberattack. And what we were going to do today is just kind of walk through how that happened and then how they responded to it and how life has been since then. So, first of all, I guess I wanted to extend my thanks. This is an extremely uncomfortable topic, and I really appreciate you not only sharing it this time, but all the times that you have shared it. And I'm sure it's getting old. [00:06:54] Speaker A: I don't know if it's necessarily old. It's just it becomes more uncomfortable, really, as I tell it. You think it become easier to tell, but there's still a lot of pain that our employees have felt because of this and the trauma that they've had because of this. So I think it's a hard story to tell, but it's a necessary story to tell. [00:07:13] Speaker D: Yeah, and I appreciate that. And I think there's a lot of, speaking of the traumatic fallout from this, if you want to call it that. I think that's probably one overlooked aspect of cyber incidents. I think when I think of a cyber incident, I don't always think of people having PTSD, so to speak, or people just kind of being like. I think the human aspect of this is a lot different than what people realize in truth, right? [00:07:44] Speaker A: Yeah, absolutely. And you think about it. If you have a cyber event to the level that we had, you're asking your employees to adopt a whole bunch of change suddenly and immediately, with no option to go back to how it was. And then certain employees are also going to get burnt out because you're going to put the brunt of recovery onto them, and it takes its soul over time. [00:08:08] Speaker D: Yeah. So we're getting ahead of ourselves. So let's just walk back to November 7. Right. 2021. And you weren't here at that time. You came post incident, correct? [00:08:21] Speaker A: I did, yeah. I joined co op about eight months later. [00:08:25] Speaker D: So probably helping with some of the recovery efforts. [00:08:27] Speaker A: I imagine from a technical standpoint, they had done really well for recovery, but what they hadn't done is the actual programmatic side, the maturity of the program, and that's where I've been able to kind of help out. I was really impressed with what they had done technically, since the incident, and I thought there would probably be more to do in that area. [00:08:48] Speaker D: That's interesting. So walk us through. I know you've got a nice breakdown. What happened that morning of November 7, 2021. [00:08:56] Speaker A: Yeah, I get the fortunate or unfortunate job of being the historian of this event. So I've talked to a lot of people with the organization, but to kind of break it down or what happened, it starts off with that kind of middle of the night. November 7 was a Sunday, so it's Sunday morning. Dispatch is having some issues. We have a 24/7 dispatch here. They can't log into a system, get kicked out, middle of the night, can't log in. What do they do? They call the it guy. So the it guy, of course, gets that call in the middle of the night, and it's one of the most painful calls to get. Weekend, you're asleep. [00:09:33] Speaker D: Sure. [00:09:33] Speaker A: They can't log in. [00:09:35] Speaker D: Sounds like a routine it ticket, though. [00:09:39] Speaker A: Yeah, that's what it starts off as. Right. They can't log in. And you have to try to figure that out. You don't know that it's been a massive cyber event. At that point. You just think, okay, what, did they forget their password? [00:09:51] Speaker D: Right. [00:09:52] Speaker A: And so the IT manager at that time tried to log in remotely and couldn't. And. Well, that's strange. And headed into the office, got into the office about 05:00 a.m. And found that they couldn't log into certain systems, couldn't log into the servers, called in for backup, help with Jay, who's also told the story with me at different events called J, in, which rarely happens. That particular manager at that time rarely called JN for that extra help, called him in, called the CTO at that point to kind of notify them that something was happening. They didn't know what. The manager headed up to our branch office up in Reed, where we have a backup data center, and as Jay was kind of poking around, trying to figure things out, he came across this unfortunate file called readme text, and it happened to be in every directory on the system. And it kind of was that. I call it that love letter from the hackers saying that we own you at this point. Give us some bitcoin and we'll release your know what they had at that point. They couldn't log into their esxi systems. They couldn't log into the NISC systems, their desktops, the files, anything they tried to open was encrypted. Everything was kind of owned. So you think about what was down at that point, and from a cooperative standpoint, phones are down, emails down, iview is down, OmS is not working. It appears that ScADA is not working. ScaDA wasn't actually compromised in this particular event, but we couldn't get to it. So you kind of have to treat it as if it's not trusted for a period of time, especially at the time of the incident, the zero day of the event, or at boom. Yeah. [00:11:40] Speaker D: Is this something that, as a co op, DMEA had at that time prepared for? [00:11:46] Speaker A: Yeah. So like most other organizations, we had a backup system and plan on, hey, we're going to have a backup, and we're going to recover from this type of event. When they developed that backup system, they were really concerned about the airport, which as you flew in, was very close, very close to our office. You practically see it right out our door. [00:12:07] Speaker D: You could almost walk there from here. [00:12:09] Speaker A: You could walk there. And they were concerned about an airport incident. Plane comes off, hits headquarters. So they moved our data or backup to our other location, which is about 20 miles away. We have a data center up there, and they had backups. And so of course they get to a point like, oh, we need to recover. [00:12:29] Speaker D: Right. [00:12:31] Speaker A: And then they went to try to access those backups, which happened to be on network. They were not immutable, and they were also not encrypted, but they were deleted. They were gone. So prior to the event, and I've seen logs of this where you can kind of see our data usage on that device up until about 5 minutes prior to the attack, and then it just went to nothing. And then those files were not recoverable the way that they deleted them. [00:12:57] Speaker D: So at this point, just to kind of like level set, we started off four or 05:00 in the morning with a simple routine, it situation can't log in. Now that we're here on site, we're realizing that it's bigger than that and there's ransomware involved. There's ransom notes on ReadMe Txt files in every directory. There's probably a level of panic that's starting to induce. Other people are involved. And now we go to look at the backups and they don't exist. Yeah. [00:13:28] Speaker A: And let me back up a little bit, too, because the story is kind of interesting for us. There's a couple of other things that happened to us. One is the Friday before this event, we had closed our office for Covid, so we had sent all of our employees home. The second thing is, we had a new CEO. Our CEO at the time, she had been with DMEA since late August. [00:13:52] Speaker D: Not even six months. [00:13:53] Speaker A: Yeah, not even six months. Just six weeks, whatever that might be. Nine weeks. I don't even know the math, but just a few months. [00:14:01] Speaker D: Wow. [00:14:02] Speaker A: And coming into this incident, and so she happened to be. I've heard her version of the story. She came out to our statewide while I was at another co op, told the story. We left with soaked pits, feeling like, oh, my gosh, we got to do more. Sure, because of how she told the story. But she was at home checking her email. 07:00 a.m. In the morning, couldn't check her email. Called up and said, hey, what's going on? Hey, you should come into the office. And what she found was our it manager was kind of in a state of shock, in a sense. [00:14:34] Speaker D: Wow. [00:14:36] Speaker A: Yeah. She kept trying to ask questions like, well, what about our backups? Well, how do you define backups? Because they're gone, right. And didn't really know what to do. And I call our ceos or ceos of co ops, the chief problem solver, because when a problem is big enough, it floats up to the top, and they're the ones ultimately where the buck stops. [00:14:56] Speaker D: Sure. [00:14:57] Speaker A: She was very resourceful, got us in contact with a third party vendor who was out in the middle of the day. His story, his version of the story, because there's so many different stories of this event. He was out helping his son change a clutch in his son's car. Takes this call, kind of finishes up the clutch while trying to talk with the CEO at that time. Says, I'll get a plane ticket out and I'll come help. You guys kept telling our CEO what questions to ask for our it folks. Did we have backups? Did we have snapshots on our san. Have we contacted cyber insurance yet? All those things really kind of helped us guide through that event. [00:15:37] Speaker D: And so I want to interject at this point. Did DMEA at that time have a cyber incident response plan? [00:15:43] Speaker A: So. We did not have a cyber incident response plan at that time. We had a policy from the board, so the board had done their due diligence, requiring us to have. It was called something different, but as essentially a recovery type plan. [00:15:56] Speaker D: Okay. [00:15:58] Speaker A: As staff, we did not have that particular document. [00:16:01] Speaker D: Okay. Yeah. Just curious, because it sounds like a lot of these recovery efforts had to be kind of done on the fly. Right? [00:16:11] Speaker A: Yeah. And there's a certain point. You said it earlier, which actually, I said it at a previous talk. When you get punched in the face, like, all your plans go out the window. Right. [00:16:20] Speaker D: Right. [00:16:21] Speaker A: When you get to a point, your backups are gone, and every system you have is ransomware. That's a huge punch in the face. [00:16:29] Speaker D: Sure. [00:16:31] Speaker A: Or gut or whatever it is. I think a plan still would have been helpful, but there's no plan for that, per se. [00:16:40] Speaker D: Right. [00:16:42] Speaker A: Where the plan comes in handy, though, is kind of after that first, initial moment when you have to start thinking about cyber insurance, contacting legal, how do you tell your members? How do you tell your employees? [00:16:54] Speaker D: Right. [00:16:54] Speaker A: And that's where a plan would have been hugely beneficial. [00:16:57] Speaker D: So I guess, too, one thing I heard you say was that the IT person or the IT manager was in shock. And I think, again, we talked about this at the beginning of the interview, but I think there's a human factor to these incidents that tends to be overlooked. Right. [00:17:13] Speaker A: Yeah. And this is my interpretation, from the stories I've heard, that he was in shock. I can't tell you for sure that that was the case, but based on his reactions and how he responded or didn't respond, and I try to put myself in his shoes, if that was me and I was in that spot and I didn't have a plan and my backups were trashed and things were in a ransom state, how would I respond to that? And I think that would be very stressful. [00:17:38] Speaker B: Yeah. [00:17:39] Speaker D: I think that would be a shocking situation. And I think with any grief response that humans have, because it's really a grief response at that point, I think you actually go through a state of denial almost. [00:17:53] Speaker A: Yeah. [00:17:54] Speaker D: And so that's a very interesting aspect of that as well. Okay, so after this, we know the backups are gone. Now the CEO is involved. We also have a third party coming in. What happened? [00:18:07] Speaker A: So let me back up a little bit, too. I like to talk about what we think happened, how the compromise happened as just a point of reference for folks. Sure. We believe that we had a vulnerability related to the exchange proxy shell. Microsoft had released a series patches in 2021 earlier in that year to help remedy this particular issue. We believe that one of our two exchange servers was, I hate to use the term, unpatched. It didn't have the patch. It's not that we didn't have a process for patching. The patching process failed. We didn't know that that server was the secondary. It became the primary. At some point, they got in through their escalated privileges by capturing a password of a domain admin. We did not have separation of privileges. So if you were an it, you had a domain admin account. That was the account you used. [00:19:03] Speaker D: Sure. [00:19:03] Speaker A: And then from there, they completely owned us. And then that first week, after you talk about the day itself, the day is a complete disaster, a mess. Things are happening. You don't know what state you're in, and you have to recover from that. Right. You have to start thinking about how do we move forward? And it's not this linear process, because your backups are done. It's not like you can say, well, I'm going to restore this backup first and get this system online. You don't know what you can trust. You don't know what data you have or don't have. You don't even know who can help. And so third party came in the next day, on a Monday. We started to contact folks like our cyber insurance. They activated their own, or an incident response team for us. Got a vendor in touch with us that would help us with the forensics with some of the recovery type efforts. Of course, you have to call your board president almost right away. On that first day, we contacted department of Homeland Security, contacted the FBI through that week, not necessarily immediately. The first day we consulted with our cyber insurance and with our legal. At that point, we also contacted NISC. We found that NISC had, one piece of luck for us is they had some data and they were hugely helpful for us. I can't thank NISC enough for what they've done for us. I have some notepads that I have in my office here, and they're special to me because they capture our CEO's notes during that first week. And there's a couple of things I've highlighted. I have one in a slide deck right here. And it said from November eigth, and it said best case for recovery was four days. Worst case was two weeks. And that was the estimate at day one before they really had a good understanding of what it would really take. It took a lot more in two weeks, of course. So when we talk about that recovery, you're trying to figure out what data, what systems you have, what can be used. We had a VM horizon client set up a VDI infrastructure that pretty much has to be rebuilt. You don't trust it. We're pulling folks in that had macs or backups of their own and trying to get their data sets. We're pulling PST files off of machines to capture emails. You kind of do all these various different things. Insurance is doing what they do best. In some cases, they interact with the bad actor. We don't go into a lot of details there other than to say you may be in a situation like ours someday, hopefully never. But if you are, you have to make that choice. Do you pay a ransom or not? [00:22:01] Speaker D: Yeah, it's a big question. [00:22:03] Speaker A: And the thing I put out there is even if you pay the ransom, you may not be able to recover your data either. It's unrecoverable, can't, there's a technical issue that comes up. They don't give you a recovery key. [00:22:16] Speaker D: Right. [00:22:17] Speaker A: There's many different reasons for it. But there's also instances where you have to make that as a business decision. Correct. So if you and your leadership team talk about that ahead of time, you'll be better prepared to be in that moment to make that decision. [00:22:32] Speaker D: Yeah. And I think the easy button here is don't pay the ransom. Right. But I think reality is, and I've talked to several different entities who have been through incidents, and when it's that business critical, like, say, for example, a hospital, there's a good chance that that ransom is going to get paid. Now, does it mean it's going to actually do what they hope it does? We've heard too many cases where it doesn't. However, I think you're right. There's a lot of pressure, there's a lot of leverage, and it's a business decision at the end of the day. And is it an investment to pay the ransom? It probably, you know, I think the easy button is no, don't ever pay it. And that's what the FBI recommends. But I think day to day living, I think is a little bit different. Right. [00:23:15] Speaker A: Well, and ultimately, business continuity is your primary objective. And if you're faced with, like, we were not being able to bill for our electric usage for the month of November and still having to pay our employees, still having to pay our power bills, you have to make that decision whether you pay or not. [00:23:34] Speaker D: So what systems were impacted at this point as far as what needed to be recovered? [00:23:41] Speaker A: So I think the easiest thing to say is what systems were not impacted? [00:23:47] Speaker D: Okay. [00:23:47] Speaker A: Because it's a shorter list. The SCaDA system, even though we operated it in a limited functionality, it was not impacted. And then second, we were able to quickly recover our mapping system. [00:24:01] Speaker D: Okay. [00:24:02] Speaker A: And those were the two systems that. [00:24:04] Speaker D: Was all that was up? [00:24:05] Speaker A: That was all that was up. So your billing was offline, billings offline, even our phone system, we had a mitel phone system in a VM that was gone offline. [00:24:16] Speaker D: Anything pretty much that lived in VMware was toast. [00:24:20] Speaker A: Yeah. Our VMware, of course, was owned. They changed the password, so it was compromised. It had to be rebuilt. Yeah. So you think about that process of collecting the meter data, sending it off into NiSC. NISC is down, so that connection is not working. Phones, of course, email, oms, all the things we use on a daily basis, and you start thinking about it even further. The step away from that, from those systems are those legacy kind of data files people have. Like, hey, 20 years ago, I made an access database to help my workflow, and I've been using it ever since. Now it's gone. [00:25:00] Speaker D: Right? [00:25:01] Speaker A: Like, file server is gone. [00:25:02] Speaker D: Right. That's a pretty deep impact. [00:25:06] Speaker A: Yeah. So of course that first week was really chaotic, but we got the glimmer of hope within IC being able to. Basically, we're working on a project with them for our subsidiary, our fiber optic subsidiary, to convert some data. And they had grabbed a whole copy of our customer information and our financial system. How old was that data? It was from mid October. So two weeks to three weeks at that point. [00:25:31] Speaker D: So not terrible. [00:25:32] Speaker A: And it took them about two weeks to get. What they did is they stood up an instance in their cloud, got it, and then basically recovered that data for us. There was some validation of that data we had to do. So it took us another short period of time into December to kind of get back into a functional state with that. Sure. But we were able to kind of resume operations in that area fairly quickly. Four weeks or know, I've seen what NISC charged us. They didn't charge us very much at all. They really took care of us as a co op. I know they've made some changes, too. They've seen a lot of ransomware on the rise. They have some recovery services that are kind of included now in some, it's pretty much mandatory. I think that's because of the mea. They say it's not, but I'm like, I'm not so sure about that. [00:26:23] Speaker D: The timing is too coincidental. [00:26:24] Speaker A: Timing is too coincidental. Yeah. [00:26:27] Speaker D: So let's talk a little bit more about the recovery timeline and how long did it take everything to get back up to good or to a good known state? [00:26:37] Speaker A: So I'd say every service that we had, since we couldn't do this in a linear fashion, we had to repair and recover, rebuild systems kind of at different timelines, different periods. So, like our phone system, we had a basic phone set up kind of within a couple of days where we had one or two lines. And from there, we moved into rebuilding the phone system. We had email, we moved straight to Microsoft hosted no need to rebuild exchange. So we did that fairly quickly. But things like the meter system integration into Ivue didn't really happen until January. During December, we had to manually move that data in between the systems. Gotcha. That integration kind of was recovered. Then moving into January, iview was kind of mid December, we got to a point where we fully trusted ScaDA, again early December, where we kind of got back to its functionality. So various points through December were things kind of getting back to a normal state. We still never recovered certain files. Like our Nic doc vault was one of those systems that got compromised that was not backed up by them, that our backup system didn't, of course, couldn't recover. And so anything that was in Doc vault was gone. [00:28:04] Speaker D: So that's all your historical data. So how many years do you think of historical data did you lose there? [00:28:13] Speaker A: I've seen news articles say we've lost 30 years of data. I don't know if it was quite 30 years of data, but we lost a lot of data. Significant, significant, many years worth. We had physical copies. We've had a process where an employee has rescanned those documents in. It's not a complete loss per se, but it's taken some time to recover it back to a quick to access state. [00:28:35] Speaker D: So all of this, in my mind, starts to build out the question of all of what you're telling me, and we're talking months here of workload for all these folks. What kind of workload was everybody under during this recovery period? Yeah. [00:28:49] Speaker A: You think about the workload and the pressure. And it, of course, sure. You have to think about it, is worried about their jobs immediately because you don't know how decision makers are going to handle the situation. [00:29:03] Speaker D: Sure. [00:29:03] Speaker A: And if they're going to show any grace or if they're just going to let you go, even if you weren't the cause or the reason, even if you're secondary, that's a concern you will have. [00:29:15] Speaker D: Yeah, for sure. [00:29:16] Speaker A: Regardless of it's warranted or not, it's there. And so the it folks worked, as Jay put it. He would come in before the sun was up and he would get home after his kids were asleep, after it was night again, and he would do this consistently. I've heard a story from the former CEO where she mentioned that he was going to come in on Thanksgiving Day because it happened November 7. Thanksgiving is two weeks later ish. [00:29:44] Speaker D: Sure. [00:29:44] Speaker A: He was going to come in on Thanksgiving Day. And she's like, Jay, I'm going to fire you if you come in today. Go spend some time with your family. And that was kind of how that went. Him and others in the IT department were working a large amount of hours. Sure. In addition to that, there were csrs that were working 20 to 30 hours of overtime per week. [00:30:08] Speaker D: Oh, wow. [00:30:09] Speaker A: There are finance people working 20 hours of overtime per week for a month plus period of time. [00:30:15] Speaker D: So not only this was a challenge from the workload perspective, but also since you're already having financial woes with no billing and your payroll is going to balloon now, so it's even more of an issue. Right? [00:30:30] Speaker A: Yeah. And we were very fortunate. We had a good cash reserve, so we didn't have to tap any emergency funds or any emergency credit. We were able to fund it with cash that we had available to us. [00:30:42] Speaker D: Very good. [00:30:43] Speaker A: I've seen or talked to co ops that say that, no, we would have to tap emergency funds or we might not make it from that. So I think that's important to note. But we did not bill in November, and even in December at that point, you think about it, by the time we get the reads in December, we essentially have two months worth of usage that we're going to be sending to people. [00:31:07] Speaker D: So no disconnects at Christmas. [00:31:10] Speaker A: Yeah. So no disconnects. We decided no disconnects. I thought that was really great. And then the cop also allowed for payment arrangements through January and February. [00:31:20] Speaker D: Okay. [00:31:21] Speaker A: Folks would have a few months to get caught up from not having that bill. So you think about your cash flow. It's not just hampered for one month, it's hampered for at least two, and in some cases that moved on to four different months. [00:31:34] Speaker D: Wow, it's a big deal. [00:31:37] Speaker A: Yeah. We were fortunate we were able to survive that. There are other organizations that get hit by ransomware attacks that do not survive. [00:31:48] Speaker B: And we've heard a lot of news. [00:31:49] Speaker D: Stories about this where folks are going bankrupt just because of the ransomware and all of its fallout. And I think that's for most people who haven't been through an incident like this. I think, again, there's a lot of misconceptions. Well, it's just a virus. It's a lot more than that. It's such a huge business disruption. I heard one particular CEO said that they would rather go through a physical storm than a cyber attack, and they'd been through both. And basically what their thought process was with the storm, at least I know what's going on. I can wrap my mind around a storm. I've dealt with storms before, but I really haven't dealt with cyberattacks. [00:32:31] Speaker A: You've got it exactly right. They've done storms before. They've trained and they've prepared and they have the equipment in the yard for that. You don't necessarily have extra switches or servers or things sitting in your warehouse waiting for that cyberattack to happen. [00:32:45] Speaker D: And maybe they should just throwing that out there anyway. Continue. [00:32:51] Speaker A: So you think about the impact to the co op. Of course, there was the immediate impact. There's a temporary reputational harm that comes from that. Sure. When I started trying to put together a presentation deck, I did ransomware co op, and of course DMEA comes up as the poster child. I was invited to speak in Iowa, one of the first engagements I did after I came here at their it conference and then later at their statewide conference. And when I was doing the it conference, I pulled up the slide with this article and one of the guys said, yeah, that's what I use to get more money from my board to help me be better prepared. [00:33:33] Speaker D: Right. [00:33:34] Speaker A: And it's not fun being that poster child. Of course, locally, regionally, it gets known there's an impact to your members. Some members. Initially the members, it seemed like from what I've heard from folks, they were supportive, they were concerned, supportive. But then as time goes along and it takes longer to recover, they're more concerned. They're scared for their data. They're angry at times they can't pay their bill. They don't want their power to go off. I saw some lingering effects of that. We finally did that data conversion project that NISC had our data for the next September. My office at the time was up front with the CSRs, and I would overhear the CSRs talk about their concerns and their fears that, hey, members are going to think we're having another ransomware attack because we're going to have to shut down for a couple of days to convert this data. And it was shut down on a Thursday. And Monday we come back online Monday at noon. It's been nine months. They're not going to think that. And at first, most of the calls were supportive. Hey, CSR's way. Answer a call, tell them what's going on. Okay, we'll call back. We're fine. But as Monday came around, we missed our deadline by about 30 minutes. So, of course, noon comes online, comes around, and we're not online. Members are calling in. And I heard some folks that were cussing at our csrs that were angry and like, you guys are having another cyber attack, right? No, we're going to be online. And of course, we were able to get the system conversion done at that point. It wasn't another cyber attack. But I think they're quick to assume that when you're having system issues. [00:35:16] Speaker D: Well, yeah, I think you mentioned it earlier. There's a lot of fear from the member side of, what about my data? What about my power? Am I going to be able to pay this? [00:35:25] Speaker A: And again, that's kind of a temporary reputational harm. We recently got the results back of a customer satisfaction survey, which that wasn't necessarily a concern of theirs or that we saw echoed throughout that that wasn't there. [00:35:39] Speaker D: So it didn't have a maybe long term lasting effect. [00:35:42] Speaker A: That's good. There's employee effect. I've heard the concerns from the employees. I have folks in my it department that say if we have another cyber event, they're going to quit. They can't do it again. [00:35:57] Speaker D: High stress. [00:35:58] Speaker A: High stress. They've lived it. They know it. They have lots of incentive to make sure it doesn't happen. And they've been phenomenal, in my mind, of doing what they need to do, trying to learn what they need to learn. But it's still there. The fear is there, right? There's employee burnout. I haven't heard anybody specifically say I've left because of that. But the folks I've talked to who have left have mentioned that the cyber event was impactful to them. [00:36:27] Speaker D: I actually was at a conference once, and one of the folks from the Eclaira or the AMI group was giving a talk at this conference, and she had said during, and she's a former DMEA employee now, but she at that time was still working for DMEA. And she had said, yeah, I have PTSD, because she was the one doing a lot of the rebuilding of that Ami data and those kind of things. And I think, again, we talked about this several times already, but there is definitely an emotional, and there's a whole human component to this that I don't want anybody to miss. Right. [00:37:08] Speaker A: Yeah, I think that's the thing that's not thought of a lot when you think about a major event, and this was a major event. [00:37:16] Speaker D: Yes, it was. [00:37:17] Speaker A: Right. It's a huge cyberattack on us. It had a tremendous impact to our business. And there's that human component people often overlook or don't think about. [00:37:28] Speaker D: I mean, people leave. You know how it is if it's uncertain at work as far as an employee is concerned. I know for me as an employee, if it gets uncertain at work, I get worried. And then I begin to look for more stable because people have bills to pay today and they still have. Christmas is still coming, and this is happening, and that's happening, and they've got the kids that need braces or whatever the case might be. So it becomes really weird at work and unstable. A lot of people start looking for something that's more stable. [00:38:00] Speaker A: Yeah, absolutely. [00:38:00] Speaker D: I think it's natural. [00:38:02] Speaker A: Yeah, absolutely. And you can't blame them. They've gone through a tremendous event. They don't want to go through it again. Maybe there were other things that were impacting their job, and this is just on top, over the edge, right? [00:38:14] Speaker D: Sure. Good point. So what would you say would be the lessons that you learned out of this event? [00:38:22] Speaker A: So we share kind of some of the basics because when we started talking more about this event, we had attended the first co op, Cybertech, November of 2022 in Washington, DC. Jay was sitting at a table with a co op. They were doing a Microsoft 365 session, and the person there said, oh, my co op will never adopt two factor authentication or multifactor authentication. I could not get them to do that. Jay turns and says, well, I have a story for. So we spoke at the second co op, Cybertech, in May because of kind of that thing, and we thought about what lessons can we share to them that are the basics? [00:39:03] Speaker D: Sure. [00:39:03] Speaker A: Things that every co op should be doing. And of course, immutable backups is where we start having a backup that can't be changed huge. Our backup system has multiple layers of immutable backups at this point. [00:39:15] Speaker D: Very good. [00:39:15] Speaker A: Starting with our snapshots on our sand, moving to our backup storage array or backup storage device, which replicates to an off site, disconnected, mostly disconnected device that also has immutability. [00:39:29] Speaker D: Right. [00:39:29] Speaker A: And then we've started to integrate a cloud based immutable option as well. So you can't have too many backups is a line of thought that we have. [00:39:38] Speaker D: Sure. [00:39:38] Speaker A: Making sure you have modern endpoint detection and response capabilities. We had older ANI virus from another vendor that just wasn't good enough to. [00:39:47] Speaker C: Stop or catch that. [00:39:49] Speaker A: Having a vulnerability management program, make sure that you're doing your patching on time, that you know when you're actually paying attention. When Apple releases, hey, we have a new major security vulnerability. Making sure that you're getting that patched in a timely manner. [00:40:04] Speaker D: Absolutely. [00:40:05] Speaker A: Paying attention more to that too, like signing up for the EISac and getting their alerts and watching things like that were things that weren't necessarily done before. Just making sure you have multifactor on everything. Everything that can have it, should have it. We upgraded our firewall to a next gen firewall, of course. User training both our end users with no before being a great program for that. And then with our it staff. I've sent one of my it staff to a sans boot camp this year. [00:40:39] Speaker D: Nice. [00:40:40] Speaker A: Make sure you have good partnerships. Like we still work with our third party managed service provider that helped with our recovery efforts and they provide us another tier of support. So if one of our iT guys can't figure something out, it goes to them and then they're there for the security response component of that based on our contract with them. But other partnerships as well with third parties, and I can't stress that enough. Co ops oftentimes think that, oh, well, when this happens, we'll just do this in house. My recommendation is find a third party who specializes in incident response, get them on retainer to help you. [00:41:18] Speaker D: Very good. [00:41:19] Speaker A: Because when you have that moment and you're that it guy who is in shock, you need somebody to help with a clear head, guide you to where you need to go for recovery. [00:41:28] Speaker B: Absolutely. [00:41:29] Speaker A: Especially if it's major. [00:41:31] Speaker D: So it sounds like with this as bad as this was, that honestly, it's made you much more resilient. [00:41:37] Speaker C: Absolutely. [00:41:38] Speaker A: I mean, we've done a lot of good things technically. Again, I can't say enough about how technically sound I felt like we were coming into. There's a couple of gaps I think we still have that we will evaluate and we will find a solution for sure, but just our way of thinking. We think differently about the co op and how to secure things. And even on the OT side, when we implement stuff, it's security first. When we do it work, we try to keep the security first mindset, not convenience anymore, 100%. [00:42:11] Speaker D: And that's where I think a lot of this falls down, right? [00:42:14] Speaker A: Yeah, I think the users push back. That's what happened here, people. Oh, that's too hard. I don't want to do that. [00:42:21] Speaker D: Right. Do they feel the same way now? [00:42:26] Speaker A: Not really. We have a lot of support from our users, but we still see users fall into the, oh, I didn't lock my computer and I walked away, or I don't really know how to tell if that's a phishing email or not. And then we've also had enough turnover that we've had new folks come in that didn't share that experience, and they don't have their story from that. And so we have to try to make sure that we kind of set that culture. And that's my thought. One of the things I observed was at some point, the two it folks that are here are going to be gone. Somebody new is going to come in over time. That's going to happen. [00:43:04] Speaker D: Sure. [00:43:04] Speaker A: And how do I build a culture here that that's remembered? And we keep this first, so as new people come in, that's just the way we do things. [00:43:13] Speaker D: Maybe it's tattoos. [00:43:14] Speaker A: Tattoos. [00:43:15] Speaker D: And never forget. Never forget. [00:43:17] Speaker A: Everybody gets a tattoo summit. I think people inside get tired of me talking about it, too. I've talked at many different places, and people have heard the podcast along the lines. Podcast. I've heard the NRAC webinar. But every quarter when we get together, as all employees, I also talk about cyber, and at least once a year, I mention November 7, as you should, and people get tired of it. But it's also that reminder, like, we never, ever want to have that happen. [00:43:44] Speaker D: You could have a commemoration day. November 7 is a commemoration day at DMEA. We survived, right? [00:43:52] Speaker A: I got to tell you, the first November 7 after the event was a little bit weird because it was my first one here. Been on the job for four months. I'm like, okay, what's going to happen today? We got good defenses. No one's specifically targeting us today. We made it through today. [00:44:08] Speaker D: One question I was wanting to ask is, how did this impact as far as other attacks after the fact? [00:44:20] Speaker A: So one of the things that will happen to you when you have a vulnerability is the bad actors will identify that vulnerability, such as exchange vulnerability, and they'll sell it on the dark web. So you could go to December 2021, you can go showdown IO and see that DMBA had a vulnerability with their exchange server. That information gets sold. Bad actors, hacking groups will buy that information and they will try to exploit it. So you will see a large number of attacks based on that. We were a target. We had our users constantly targeted. With phishing, there was a high elevated number for a long period of time after the attack. And even after the attack, we had folks that would sometimes fall for phishing and put their credentials in somewhere where they shouldn't. We were better prepared to handle that, to make it a non incident versus a crisis. But you will be targeted for attack prolonged period of time, because your data is out there as being compromised. [00:45:19] Speaker D: Again, I think this is another one of those fallout pieces that most people don't take into account. They take into the account. Well, we'll just be able to recover because we've got an incident response plan. [00:45:28] Speaker A: Or what have you. [00:45:30] Speaker D: But what about additional attacks? What about now employees are leaving or bringing new employees in. What about the emotional impact of this? There's a lot to be thought about here. [00:45:38] Speaker A: Well, and think about if we were able to recover, but we didn't fully understand why we got compromised to begin with. What good would recovery do when the week later somebody comes in through the same method? [00:45:49] Speaker D: It's a very good point. It's a very good point. [00:45:52] Speaker A: Yeah. [00:45:52] Speaker D: Well, we've really talked for a long time, so I don't want to take up too much more of your time. I really appreciate you coming on the podcast. Any final thoughts that you would want. [00:46:00] Speaker A: To share with the know? I actually just want to say thank you, John, for, first off, putting attention on topics like this, giving us a safe space to talk about it, to share our story. And just for all the work you've done in evangelizing for cyber, for co. [00:46:15] Speaker D: Ops, I love co ops. [00:46:18] Speaker A: So do I. I always say they're the best jobs in our communities, 100%. [00:46:23] Speaker D: I love working with co ops. Some of the best people I've ever worked with or have been in the electric co op space. And no, greatly appreciate you coming on and just sharing your story. I mean, you have to be kind of vulnerable to share it and then, like, you know, the more you tell it, the more difficult it gets, it sounds like, yeah, I thought it would. [00:46:44] Speaker A: Be easier because I wasn't here. I'm detached from it the first time I spoke with Jay and saw the trauma that he kind of went through retelling his story, that was really hard. But as I've kind of been here longer, taken more ownership, taken more pride with it, it is a little harder to tell. [00:47:03] Speaker D: Sure. [00:47:04] Speaker A: It's like, wow, this really happened. This impacted this person. When I first told it, I didn't know necessarily impact to the csRs. Now I do. And how it impacted finance in our warehouse and our linemen and everybody else. [00:47:17] Speaker D: Well, thanks again. I guess just say that. Final thoughts that you want to share or any resources or anything of that nature? [00:47:27] Speaker A: I think there's lots of resources out there to folks, so you're not alone. Don't be afraid. Don't take no for an answer when it comes to securing your co op. And if there's anything I can do to help out, please let me know. Reach out. [00:47:43] Speaker D: So we'll include Bob's information in the show notes and be more than welcome. I know he's pretty much available to come and talk to your Statewide or your co op. He doesn't always like to do that, but I know that he will take one for the team sometimes and come and talk to. So. [00:47:59] Speaker A: And there's always webinars, too, or Zoom or whatever, if we can talk that way, too. [00:48:04] Speaker B: Absolutely. [00:48:04] Speaker D: Well, thank you so much, Bob. I greatly appreciate it. [00:48:07] Speaker A: Absolutely. Thank you for having me on. [00:48:09] Speaker B: Yeah. So, Anthony, powerful stuff, right? [00:48:14] Speaker C: Yeah, that was very powerful. John, there's a lot of takeaways from that. Know Bob did a great job. You did a great job interviewing. [00:48:24] Speaker A: Yeah. [00:48:26] Speaker C: Just starting off with the security first, know, that's really what we all have to mean. [00:48:34] Speaker A: That's not just it. [00:48:35] Speaker C: That's the whole co op employees for sure. [00:48:40] Speaker B: And I think he made a couple of good points. Several good points, actually. But one of the things that kind of stood out to me was we think differently. That's where we're trying to get as it professionals, as cyber professionals. That's the goal of what we're trying to do with people, is get them to think differently. [00:48:59] Speaker D: And I think by default, their whole. [00:49:01] Speaker B: Company thinks differently now. Right? [00:49:03] Speaker D: Yeah. [00:49:03] Speaker C: Unfortunately, it just took a big event like that to change everyone's mindset, which is my frustration. [00:49:09] Speaker B: Right. I think they get blamed as the chicken little of the world sky is falling, but really we kind of understand. I think when you go through enough training, when you've been through enough things. I've been doing cyber for 14 years. I've seen a lot of different things. I've been through some cyber incidents. I've experienced my own cyber incidents. I've talked to tons of people. You kind of know what some of this stuff is all about. And I think, like I tried to bring out in that interview was, this isn't a technical problem. I mean, it is. Obviously there's a big bunch of technical problems. A bunch of stuff is down. But did you catch how much of that is a people problem? [00:49:46] Speaker C: Yeah, well, that and just the way it impacted folks. You always think about how it impacts it folks. Sure. And we all have kind of an understanding being the it world, putting out fires, but I can't imagine putting out a fire for a month or two. But hearing him discuss how basically the employees and the membership, they have PTSD. [00:50:12] Speaker B: And people are leaving. You got good people. You've invested all this time and money in people and they up and leave. They're like, we can't do this. We can't do this. I don't know. What would you say to an it employee that said, you know what? If we have one more thing happen, I'm out of here. [00:50:28] Speaker C: Well, the problem is it's so unpredictable. I just relate it to storms. We're over on the coast and we deal with hurricanes. And if I were a 30 year employee and thinking about retiring, I would probably retire before hurricane season. I can tell you that. You just can't help that. There isn't an answer for that. [00:50:49] Speaker B: You know, when ransomware season is by chance, I got it's. There's never a not in know just. Well, Bob's a great guy, right? Like, I had a really good time. I was out there for about three days and spent some time with him out there. Of course, I love Colorado anyway, too, so that was always a good thing, but spent some good time. And Bob is just a very humble leader. He's done a very good job with his team. I did some incident response training with them recently and I was so impressed with their team. But like I said, I think there's a resilience here. That one example that came to my mind. I don't know if you've heard this story or not, but the Ukrainians, they're getting attacked 24/7 right now by Russia. [00:51:35] Speaker D: Right. [00:51:35] Speaker B: And especially their cyberattacks too. Right. These are not just kinetic attacks. And I guess that they're to a point where, and I don't remember the exact numbers, but they can tear down and set up an entire substance. I think it's 24 hours. They can do an entire substation from scratch in 24 hours because they're just that resilient because they just keep getting hammered. And I did a team building exercise with these guys as part of the incident response training that I do, and I've never seen this happen before. I said, well, here's the instructions. And I told them or whatever, and they just automatically, without even talking to each other, started breaking the tasks down for each other. And they did that one part, and then the guy said to the next, to the lady, she said, okay, now it's your turn. And they just handed it off across the entire team. They never even talked about it, and I never told them to do it. It was the most amazing thing to see. I was so impressed. [00:52:26] Speaker C: That is amazing. A lot of times people come across stuff that it's overwhelming, whether it be a cyber incident or just anything in life. And that's the perfect solution right there. If something's too big, break it down into smaller parts that you can handle. That's awesome that they have that instilled in them now. [00:52:43] Speaker B: But, you know, this is a military veteran. That's not going to happen unless they've come under fire. You know what I mean? That's the only reason they're that level of resilient. I don't know. The whole thing is just super fascinating to me. Again, thank you, Bob, for sharing that story. Again, I know you've shared it a bunch. I hope we know justice to the story on this side. I know this is a much longer podcast than normal, about double our normal length, but I felt it was necessary. [00:53:07] Speaker D: Right? [00:53:07] Speaker A: Yeah. [00:53:07] Speaker C: It's awesome that they're sharing that with us. And, folks, if you go back ten years ago or however many years ago, folks weren't really thinking about this except for the it folks, right? And with people sharing these stories like Bob, this is catching around, and you got ceos and board members that they're starting to think this way, and that's important. It's really people like Bob and Delta Montrose as a co op as a whole, sharing the story that's helping that. [00:53:35] Speaker B: Absolutely. Absolutely. All right, well, that's it for this podcast. Just a have I didn't mention this earlier, Anthony. I guess we should tell the listeners that our next episode, just to kind of give a little preview, will be one of our new sponsors. [00:53:51] Speaker C: Awesome. [00:53:52] Speaker B: Yeah. So I won't let the cat out of the bag yet as the name of that sponsor, but they are a technology startup, but a pretty cool one at that. And we're actually going to feature them on the next episode, episode seven we'll feature them for the entire episodes. We're going to allow them to kind of take over our space and give a little bit about what they do and talk about it in this cyber context. We're still looking for more sponsors. I still got about four other ones that I'm talking to. And of course, you can always donate to the podcast. We're all about that. We do have a link available for you for there. Again, like this show. Subscribe to the show, get the updates. You can do Apple. You can do Spotify. Let's grow this thing. Share it with anybody. Please share this episode, too. This is a very important one, so please share this with board members. Share this with whomever. Cut it up, dice it up, do whatever you want with it, but just get it out there, right? [00:54:44] Speaker D: Yeah. [00:54:45] Speaker C: And also, if you appreciate the show, please leave a review that helps us out. [00:54:50] Speaker B: Yeah, for sure. Absolutely. I think Anthony and I are both on the same page here that we want to get some education in people's hands. That's as real world as possible. He's still sitting in that seat. I sat in that seat for ten years. We know where you're at. You know what I'm saying? Oh, one thing I did want to mention, too is I'm going to try my very level best to be at a few conferences going forward, maybe have a booth. We actually talked about doing a live podcast from one of the conferences coming up in the near future, so we'll see how that goes. That's definitely an idea that we're tossing around here. I'm working with a sponsor on that to maybe get a booth. I'm definitely talking about doing some presentations coming up at some of these conferences and stuff, so spread the word. We just want to help. That's all we're trying to do. And we appreciate every one of you that listen. We've heard tons of great feedback. I probably got a handful of feedback over the last two weeks from folks that was just kind of coming out of the woodwork and just saying, we love the podcast. We love the podcast. Keep up the good work. And so we're doing that. That's what we're doing. Final any final words or thoughts from you, Mr. Anthony? [00:56:02] Speaker C: Really just want to thank Bob again for sharing this. But I urge you to share this story with your CEO, with your board of directors. This is important for folks to hear. It really is. [00:56:13] Speaker B: No, I absolutely agree. Well, that's all we have for off the wire of this episode. Tune in again next week or one, two, weeks rather and check out what we got. We always try to get something cool for you. If you have any ideas you want to hear anything on the show, any suggestions, please reach out to Anthony or myself. We'd love to take your feedback and make something that's going to make sense for you. So please do what you can to help us and what we do all we can to help you. And with that, I'm going to sign off. We'll see you later. [00:56:41] Speaker C: All right, thank you everyone. [00:56:43] Speaker A: One, two, three. [00:56:44] Speaker B: Come on. Thanks for listening to off the Wire, a play by play on cyber issues featuring Anthony Kent and John Watkins. Make sure to, like, subscribe, follow, and hey, share this podcast if you liked it. We appreciate it. Appreciate your time. See you next time. The views and opinions expressed in this podcast do not necessarily reflect those of John Watkins consulting or its affiliates. Always consult with a qualified cybersecurity professional for tailored advice.