[1:15] Securing the Flow

Rachael: So I'm so excited because critical infrastructure right now is such a hot topic. And we have an awesome guest today, David Travers. He serves as director of the Water Infrastructure and Cyber Resilience Division at the US Environmental Protection Agency. He manages a team of engineers and scientists. Heprovides tools, training, and direct technical assistance to the 152,000 drinking water systems and 16,000 water systems wastewater systems in the US. 

David: Hi. Thank you so much for the opportunity to discuss the important work we're doing in protecting the water sector from cyber attacks.

Rachael: Absolutely. Audra, let's kick us off.

Audra: I wanted to kick off you giving a little bit of an education to our audience about why water infrastructure systems are getting targeted by ransomware.

David: In terms of the premise of your question. I would say that the majority of cyber attacks against water systems do involve ransomware. As I'm sure you and your listeners are aware, can be highly disruptive to utility business and communication processes. But usually does not interrupt water service. The attacks are commonly executed as we've observed through phishing and social engineering schemes among other reasons, which probably makes water systems not unique. Those are the common vectors of access. 

In terms of that is an excellent question. I think the appeal of the water sector, is the common recognition that water is critical to any community. It's hard to imagine any small city ranging from a large town or going back to a rural area being viable in the absence of drinking water. When you turn on your tap, you expect water. Whether it's for drinking or bathing, brushing your teeth, whatever.

 

Navigating the Torrent

David: And then similarly on the down riverside, so to speak, is the wastewater services, which are equally important. So I think the majority of ransomware attacks are just opportunistic. I think they involve just criminals and others intending to do ill. They are exploiting vulnerabilities across all their 16 critical infrastructure sectors just across the sectors looking for opportunities to make money. So it's hard to say to what degree those ransomware attacks are directed towards water systems. which as I said, have an appeal because they're so critical to any community. 

However, the majority of ransomware attacks are likely random in nature and opportunistic. Now, we know that a number of ransomware attacks originate in Russia with either CACI support or explicit support from the Russian government. We know from public reporting from the Department of Homeland Security that Russia has targeted what's known as trying to pre-position themselves on critical infrastructure. So it ranges from, like I said, these opportunistic criminal enterprises to state-sponsored intrusions. So I hope that sort of answered your question.

Audra: It does. The one thing though, in looking at this area. Recent months have actually brought about a significant increase in ransomware attacks on the water sector. Is it just because they're out for the cash and they're just sending to anyone and there's the opportunism? It's just purely you're getting more people opening emails they shouldn't and clicking on links in the industry. Or is it more than that? Is it now more focused on the water industry?

 

The Surge in Cyber Attacks on Water Systems and the Evolving Landscape

David: Yes, that's another great question. I mean, various respective sources in the intelligence community. Just to the premise of your question again, have definitely attested to the increased frequency of cyber attacks. I've seen estimates anywhere from three to a sevenfold increase relative to where we were just a couple of years ago. But if I could, I'd like to address kind of the evolving nature of these attacks. But before we do that, I think the first thing to understand about the prevalence of cyber attacks. 

Not only in the water sector but potentially in other sectors as well. We as a nation are aware only of a modest fraction of total cyber incidents within the water sector. And as I said across other sectors as well. Water and wastewater systems are just not required to report cyber incidents, at least by federal law. There might be state laws that have those requirements, but probably only a handful.

And so many systems do not report such incidents. This is important because we are aware of the federal government being of just a small proportion I think of the total number of cyber attacks against critical infrastructure. So when we cite these numbers such as I just said, three to sevenfold increase. It's almost anecdotal in nature. But yes, there has definitely been an increase. I think it involves the lucrative nature of it as you cited in the question. As more people have come to recognize the money that can be made. The use of cryptocurrency as the common currency used for extracting a ransom from victims. 

 

Understanding the Vulnerabilities of Water Systems to Cyber Threats

David: The increased accessibility of that. And then you layer in on top of those things, which again, out of the more opportunistic criminal enterprises. These state-sponsored intrusions account for some proportion of attacks as either state-sponsored or state-endorsed. Or the state itself might be looking to probe critical infrastructure sectors for vulnerabilities. This actually is a segue to the fact that the water sector we believe is highly vulnerable to cyber-attacks. Because of the prevalence of use of operational technology systems in the system across the sector and the vulnerabilities within many water and wastewater systems.

Rachael: I mean, just for context. David too, I think for our listener's sake, I was reading somewhere that today 97% of the private water systems are serving 10,000 or fewer communities. Is that accurate?

David: Yes, it's 97% of public water systems. Which are those hundred 50,000 or so systems you mentioned at the beginning of the podcast serve fewer than 10,000 people. So the majority are smaller systems.

Rachael: Right. That's a really good contract.

Audra: What I wanted to know is if you could share any real-world examples of any state attacks as far as we're aware. Or cyber attacks that have actually impacted surrounding communities?

David: Sure. I think in terms of state attacks, you have to go international to some degree in that we are aware. I think you mentioned Rachel, an incident in Italy, there have been state-sponsored attacks. This is timely against Israel, specifically Israeli water systems. Domestically, again, we know that states such as Russia and China, perhaps Iran, and North Korea. They are seeking to pre-position themselves on our critical infrastructure, which means that they would have access to the operational technology of the water systems. 

 

[10:33] Safeguarding Water Systems from Cyber Threats

David: And sorry if I'm using a term that's maybe unfamiliar to you or your listers, let me know. Operational technology is simply the hardware and software that are used to control virtually all aspects of water system operations from pulling water. For example, from a river or a well to treating it and then pumping it out and storing it in the system. 

So there are opportunities to disrupt those services, which as you could imagine would have. We take water for granted in this country. And you can imagine if there was a disruption of that and it was attributed to a foreign state. That would have an enormous impact on the public psyche and maybe our willingness to engage with these state adversaries.

So yes, it's a significant concern for the water sector which is why EPA offers, which I'm sure we'll get into in a little bit. Just an array of tools and technical assistance to help these water systems defend from those types of advanced persistent threats as well as internal threats. I will say there was since you asked about specific incidents involving cybersecurity in the water sector. A notable one that comes to mind is an incident in Kansas where an employee was fired then the system neglected to revoke that individual's remote access credentials. 

The person hacked back into the system. Well, didn't hack back in, actually just used their existing credentials and took the water treatment process. The treatment trains down disrupting service to that community. And there are a number of incidents. There was another one that was revealed a few weeks ago involving a system in California where there was a contractor working for a utility. 

 

Guarding the Lifelines

David: And that individual, after retiring not being fired this time just retiring, decided to uninstall the operating system at the water system, which again, controls pumps, treatment, et cetera. The things that are needed to operate the water system. So, unfortunately, there are many examples of cyber attacks either potentially causing disruption or disrupting water services in the US.

Rachael: And I think the California example, you mentioned David, he had installed that while he was working there. This is playing the long game and then time goes by, he leaves the company, oh, you know what, lemme just go and execute that. That is crazy.

David: And I hesitate to speculate on his motivation, but you are correct. He did install a kind of software on his own private computer that would later enable him to connect to the water systems computer. This is in Discovery Bay, California Water System and proceed to uninstall the system. Fortunately, the system managed did not compromise service because they responded well to the incident. But those incidents are clearly our concern.

Rachael: Yes, absolutely. 

Audra: So considering the state of operational technologies and maybe vulnerabilities within them. What are we doing to actually help protect the water companies? What kind of safety nets are there? How are we helping to proactively start removing some of the vulnerabilities? Or being able to alert if someone goes in and tries to uninstall an operating system or things like that, what kind of things are in place?

 

Strengthening Cyber Resilience in Water Systems

David: Sure. And I think it's worth noting since we've been talking about these incidents. It's worth noting here a recurring theme. These incidents have made clear that many water systems unfortunately have not implemented basic cybersecurity best practices such as software patches. And network access controls as we were just talking about. And that these gaping security deficiencies leave water systems vulnerable to these potentially disabling cyber attacks. So about EPA, I'm not sure if you or your audience is aware of this. 

However, the federal government has identified these 16 critical infrastructure sectors such as water, healthcare, emergency services, energy, and so forth. Each of these sectors has something called a sector risk management agency or SRMA, which serves as the federal lead agency responsible for enhancing that sector's security and resilience against all hazards. Whether malevolent acts like we've just been talking about, natural disasters, climate change, and so forth.

And so by this presidential directive, EPA, which is why you're talking to me. This serves as this sector risk management agency for the water sector, which includes both drinking water and wastewater systems. And so one of the things that we do as an office, particularly with respect to cybersecurity is provide critical tools and training for the sector. So just one example is we offer the cybersecurity evaluation program whereby EPA will have one of our contractors conduct a cybersecurity assessment at a utility and then provide that utility with a risk management plan so that the system understands, hey, what are the significant cyber risks facing my system? And what are the measures that we can implement to mitigate that risk? 

 

Simplifying Cybersecurity for Water Systems

David: So that is an important program that we offer, particularly for smaller systems that might be too daunted to even begin a cybersecurity assessment. Because when you think of cybersecurity, you think, oh gosh, this is going to be hopelessly complicated and incredibly expensive. And part of our mission at EPA is to convince water systems that cybersecurity need not be either of those things that through a relatively simple list of questions and through largely procedural changes, you can significantly mitigate the risk of cybersecurity. 

So we just talked about the individual who was fired or the contractor who got back into the system. So a common procedure could be to revoke the access credentials of individuals who no longer work for the utility. It's not implementing some crazy expensive mitigation measure that would cost tens of thousands of dollars a month. But sometimes just procedural matters. So that is an important part of what we at EPA do. We offer a variety of services, and I can certainly go through all of them. But I think it would be pretty excruciating if I did. But there are other tools and things we call tabletop exercises where we walk utility through. 

You've just lost the use of your operational technology system, and your ability to control your system remotely. What will you do now who you contact internally and externally to utility to help you out? So we take our mission with respect to cybersecurity very seriously. Fortunately, we're not alone because the cyber security and infrastructure security agency over at the Department of Homeland Security also offers a great deal of services. 

 

[19:08]Building Cyber Resilience in Water Systems Through Education

David: And we often partner with them in promoting cybersecurity in the sector. We partner with owners and operators in the sector and their associations to promote tools that the water associations have. CSA has, EPA have to let them know that collectively we are responsible for improving cybersecurity in the country.

Audra: So considering that ransomware generally is dependent on someone doing something. Someone has to click on something, someone has to open something. Then that can be human error, that can be all sorts of different things or curiosity, it could be known as well. What kind of support are you offering to help people become educated on this? Because it's going to be employees that actually kind of form the first line of defense when it comes to dealing with ransomware.

David: That's completely accurate in that even the most sophisticated water system with a billion-dollar-plus budget. Many employees and an excellent cybersecurity program can be a victim if not all of its employees are aware of the threat of cybersecurity. So what we encourage at EPA and what a water association CI encourages is to conduct training of staff so that they understand that again, we have this collective responsibility to protect these critical services from cyber-attacks. And every employee needs to play a role in that. 

I think that that is particularly important because especially with ransomware attacks and phishing. I'm just going to pull back a little bit and say that as ransomware attacks. They generally target, the IT or business enterprise side of utilities, water utilities, and other victims. So for water utilities, that means things like billing, accounting, maybe their internal email systems, customers, and personal information where you live. 

 

Unraveling the Interconnected Challenge of Cybersecurity in Water Systems

David: And how much you're being billed for water ever since the first notable ransomware attack? You may remember, and your listeners may remember the WannaCry incident in 2017, which was perpetrated as we understand it by North Korea. IT systems have borne the brunt of ransomware attacks. And this is a really important point. Recent incidents at water systems and high-profile attacks such as you may remember the Colonial pipeline incident. The JBS meat processing incident, they underscores how an attack that can be initiated on the IT side of the house can affect the operational technology or the OT side. 

So as we said, for the water sector. The OT side would mean the ability to deliver and treat safe drinking water or collect and treat wastewater so that even cyber attacks target IT systems and maybe originate with ransomware. They can adversely affect OT systems. So I'm mentioning this because just raising this awareness that you may be an employee in the accounts department or HR. But if the system is not sequestering, it's IT from its OT systems. And we've seen this again in the water sector. You can have an impact on the water system operation. So training and awareness are critical components of any water system cybersecurity program.

Rachael: Definitely. Can I ask a quick question too? It seems that this connectivity to the internet is a problem. Is there a world where we ever go back to manual where we just come offline and mitigate the threat that way or I know that sounds crazy? But you kind of get to a point where there are just so many vulnerability vectors. Is an answer to take some things offline and keep others on, or I just try to get creative problem-solving.

 

Navigating the Integration of Technology and Security in Water Systems

David: I understand, and one of the basic countermeasures we recommend to utilities. This is not available to all utilities, but we believe it's available to most. I mentioned this when we conducted our tabletop exercise, can you operate your water system manually? Can you operate it without your SCADA system as it's called, your Supervisory Control and Data Acquisition system? Can you operate your system without these internet-facing devices and without the operational technology that hardware and software that you would ordinarily use to operate your system? 

So it is, you are correct. It is an important part of having a cyber emergency response plan, which is something we encourage all utilities to have. Now, having said that it's 2023, IT and OT systems are so firmly embedded in how we operate. Just virtually every aspect of critical infrastructure. Think of your car, right?

I have an EV, I wouldn't say it's frightening to see how much that car is dependent on the internet and the possibility of it being hacked. But it just pervades every aspect of our lives. I'm sure we all have iPhones or equivalents and are dependent on them. Similarly, water systems have relied on or do currently rely on operational technology and IT systems to operate their systems efficiently. 

You have to think that a lot of these systems are dispersed over in some instances hundreds of miles. And so to operate their system, it's important to pull in information from the treatment plant. From the intake. Are the pumps cycling on and off? How full is the storage tank? All this information needs to be obtained, accumulated, and assessed. So that the operator can ensure that the water system is meeting the needs of its customers and both quantity and quality.

 

Safeguarding Water Systems in the Digital Era

David: So in other words, this technology has been so firmly embedded in how water systems operate. It's hard to extricate it. Now, you could during an emergency operate on manual, but long-term, it's really unsustainable. And a lot of the smaller rural water systems, we mentioned those earlier in the conversation. Sometimes you'll have a single operator responsible for multiple water systems just for cost savings purposes. 

These are not generally incredibly wealthy communities and if nothing else, they don't have the economies of scale to have an operator for their plants. So they might have one operator for multiple plants. And the only way that's feasible is if that operator has remote access to those water systems because they need to know. Hey, is the chlorinator functioning or the wells functioning? Are the tanks full? That sort of thing. So I think that the sector has embraced rightfully this IT and OT dependency. But because of that embracing of the technology, there's an equal responsibility to ensure that that technology is protected. 

Again, there are a lot of procedural steps that can be taken to protect that technological infrastructure without forsaking the infrastructure entirely, which would create all sorts of challenges for operators. Hopefully, my rambling response provided you and your listeners with an understanding of how important that infrastructure is to water systems.

Audra: So thinking from a wider perspective, ransomware is obviously a problem. But the wider cybersecurity issues that these operators face, have they ever considered going air gap and actually kind of splitting up their networks to give a true separation from IT to OT to try and increase security from that perspective?

 

[28:23]Bridging the Gap

David: Yes. It's like you're reading from our playlist here. Another basic measure we recommend is having a separation between the IT and OT systems. The challenge that we found is not all water systems, in fact, potentially the majority of water systems have not conducted an inventory of their IT and OT systems. Which means there's not always an understanding of how their IT and OT systems are interconnected. 

So they need to take that basic steps. Step before air gapping or sequestering the IT from the OT side. Because what we found is that, some of these ransomware attacks originated on the IT side. And then, the water system discovers the IT side controls all our communication at the utility, which in turn controls our ability to collect information from throughout the system. So we had an incident. A couple of incidents I remember in Pennsylvania and Ohio where there was a ransomware attack on the IT side. And that prevented the water utility from receiving alarms from the pumps at the wells.

So they had to send operators out there to do it manually. It prevented another utility from using kind of a central place to remotely access its treatment system. So they actually had to have operators on site running their water system as opposed to, as I said, sometimes operators like to do that or need to do that remotely. 

So yes, that is another important component of a cybersecurity program. But a lot of water utilities need assistance in even getting to the point where they can sequester it. They don't even know what assets are, in fact, internet pacing, and they don't know how their IT system is actually connected to their OT system either.

 

Cultivating a Cybersecurity Culture in Water Systems

Audra: So is this where you randomly find a firewall somewhere in your network that you didn't know was there? I experienced that a long time ago.

David: Yes, no, I'd love for them to randomly find firewalls because oftentimes, well, I can't say oftentimes. But it's not infrequently that we see that those firewalls don't in fact even exist. So yes, the issue that we're struggling with and that maybe I'm dancing around is really the water sector has, I think struggled to really by its own admission. They conducted, the industry, conducted a survey of itself in 2021, and the number one impediment to a drop in adopting cybersecurity programs. 

Even at larger systems, these are systems serving over a hundred thousand people is simply the lack of cybersecurity culture at the utility. It's just not utilities are dealing with, you think of your average utility, they're dealing with aging infrastructure. Some of the pipes in our countries are well over a hundred years old and deteriorating. They're dealing with demographic changes. People coming in or leaving the area, expanding or shrinking, losing or gaining economic vitality in the community.

They're dealing with regulatory compliance, ensuring that the water meets safety standards, they're dealing with all sorts of workforce issues. So layering cybersecurity on top of that can be a challenge for a lot of water utilities. And some of them are like me, right? When I went to graduate school and learned how to design water and wastewater systems, cybersecurity wasn't part of my education in my fifties. 

 

Nurturing a New Culture to Safeguard Water Systems

David: A lot of people in the WHO who operate water and wastewater plants are probably of a similar age and level of experience and wisdom as I am. But the downside of that is they didn't necessarily learn cybersecurity as part of their education. And so it's considered not an inherent part of operating their water systems. And it's that culture that EPA working with the water associations, owners, operators and SISA are trying to change. 

It should be part of the culture just as you put a padlock on your storage tank so somebody can't open it up and throw something in there. You need to do the same thing on the IT and OT side as well. And it has been something of a challenge that we're working collectively to try to overcome. But that lack of a cybersecurity culture is definitely challenging because a lot of water systems don't move off of step one to assessing cybersecurity vulnerabilities and then of course doing something to mitigate them.

Rachael: Can I ask a question then too, where we look at the attraction of critical infrastructure and particularly water systems for malicious attacks? Are we at a place where you have to have your ransomware budget? Is that becoming kind of part and parcel of how they operate? I think a lot of these gangs, right, they'll take a look at, okay, well here's your operating budget. Here's probably what you could afford. If you're rural, maybe it's $20,000. If you're larger like New York, maybe it's a million dollars, I don't know. But is that just having, is that a pat line item for budgets ahead for water systems that need to start thinking that way?

 

Investing in Resilience

David: Well, I believe the official policy of the United States government is that individuals should not pay ransomware. So I will not deviate from that. However, I would encourage water systems to look at what they might be paying in ransomware, whether it's tens of thousands or hundreds of thousands of dollars. I believe the city of Baltimore was struck by a ransomware attack, and I think the actors were asking for maybe $600,000. It was a sizable sum of money and the city ended up declining to pay for that. 

As a result, they had to spend significantly more than that on essentially restoring its IT services in the city. And so what we're encouraging is don't set aside money for ransomware, set aside a budget to support a cybersecurity program so that you are not a victim of ransomware in the first place. Now, water systems can also, and other critical infrastructure owners can buy actually by insurance related to ransomware. But that insurance, because of the sheer prevalence of ransomware is becoming increasingly expensive and increasingly requires that the policyholder have basic cybersecurity practices clearly to reduce the chances that they'll be a victim. 

So even if you go for insurance and don't want to adopt a cybersecurity program, an insurance company is likely to require you to adopt some sort of cybersecurity measures. So yes, use that ransomware money that you might set aside to actually invest in prevention as opposed to paying money to a criminal enterprise.

 

[36:19]From Candy Dreams to Cyber Resilience

Audra: I think that's a very good recommendation. So David, one morning when say you were seven years old, did you wake up and think, I know what I want to be when I grow up? I want to be director of water infrastructure and cyber resilience divisions at the US Environmental Protection Agency. Did that happen or has it been a different kind of path?

David: So it was probably a different path insofar as when I was seven. I was probably thinking about how I could get money from my parents to buy candy. That was probably the extent of my intellectual capability back then. My path to where I am today is a somewhat torturous one, and I wrote to regale you with the details. But yes, it is hard to say. I actually started college as I was a classics major at my alma mater, the University of Chicago. 

So was intended to be a historian or archeologist, but then realized I should probably acquire the skills to actually make a living. So I picked up a master's of public health. I'd spent a gap year working for the United Nations in Nairobi, Kenya, and working on water and wastewater issues and just became enamored with the problem of adequate water. Something that I've spoken about this before, earlier is something we just take for granted in this country.

 

Navigating Waves and Cyber Threats

David: So I went to graduate school, picked up a master's in public health, and later a PhD in environmental engineering. And then with my interest in water systems and wastewater systems also had maybe an antiquated sense of public service. So thought that working for the federal government would be an excellent opportunity for me to bring my expertise to federal policy and perhaps make a difference on a national level. 

And so I started out doing assessments of drinking water infrastructure under this multi-billion dollar program we have at EPA called the State Revolving Fund, which gives money to water systems to improve their water quality and their operations. And then after 9/11, I pivoted to working on homeland security issues. Initially, we had a focus on physical terrorism, understandably given the trauma of 9/11 on the nation. And then we had the 2005 hurricane season.

So we shifted to, oh, we need to increase our understanding of threats to include natural disasters. And then there's a realization that climate change could affect water systems in terms of drought and the opposite flooding and so many sea level rise. Of course, we had COVID, which affected the supply chain, really disrupted the supply chain, and that included the supply chain for water systems. And then thrown in that mix was cybersecurity, which became another of the threats that we had to deal with. And so that's how I became vested, I guess, in the whole notion of cybersecurity for the water systems of this country. So as I said, a fairly torturous path, but that's how I arrived here.

 

Public Service Odyssey in Safeguarding Water Systems

Audra: Sounds fascinating. A very noble direction. 

Rachael: Yes, absolutely. And I love this idea of public service too. I started out that way, and it reminds me too of our conversation, Audra, with Josh Corman. He stood up this foundation called I Am the Cavalry and the power that one person can have when they have a passion or a conviction for something. And particularly being in government, how do you change things unless it's from the inside out? So I applaud your public service because it's so critical if we're ever going to make any meaningful changes.

David: Thank you. I appreciate that. And I hasten to add that I am only as, I guess, influential or impactful as my colleagues within the division. That's where the actual expertise on things like cybersecurity emergency response and contamination events lies. So the good news is it's certainly not just me with a public commitment. My colleagues within the division have a similar level of commitment. 

And it frankly makes me feel proud of the degree to which the federal government has managed to recruit these excellent individuals who work for all of us and who are determined that in the case of my division, that water systems continue to operate no matter what the challenge. So hopefully these are interesting times we live in, but hopefully, that provides some reassurance to you and your listeners that there are these individuals a hundred percent committed to the task at hand, which is protecting wastewater and wastewater systems from all sorts of hazards.

 

Guardians of the Stream

Rachael: We appreciate it. We all take it for granted. My grandma used to have a well-out back, and I won't tell you what that was like when we tried to wash clothes. I love my water, I love my city water, and I'm very grateful. I'm very grateful it takes a village. Well, David, thank you so much. I love learning so many new things. That's why I love cybersecurity. You get to learn something new every single day. And thank you for sharing all these amazing insights with our listeners.

David: Yes, it was my pleasure. Thank you, Rachel. Thank you, Audra. I really appreciate the opportunity to talk to you and your listeners today.

Rachael: Fantastic. So to all of our listeners out there, thank you again for joining us. And as always, don't forget to subscribe. You can get David's episode directed right to your email inbox every Tuesday. So until next time, everyone. Be safe. 
Thanks for joining us for the To The Point Cybersecurity Podcast brought to you by Forcepoint. For more information and show notes from today's episode, please visit forcepoint.com/gov podcast. And don't forget to subscribe and leave a review on Apple Podcasts, Google Podcast, Spotify, or Stitcher.

 

About Our Guest

 

Director of EPA’s Water Security Division in the Office of Water, Dr. David Travers manages a team of engineers and scientists in providing tools, training, and direct technical assistance to the 152,000 drinking water systems and 16,000 wastewater systems in the US. Each year, the Water Security Division trains over 5,000 water/wastewater utilities, state/tribal officials, and federal emergency responders to become more resilient to any natural or manmade incident—including cyberattacks, climate change, hurricanes, drought—that could endanger water and wastewater services. Prior to David’s current role, he directed the Drinking Water Infrastructure Survey which assessed the current and future capital investment needs of drinking water systems. David has a PhD in environmental engineering a Master of Public Health from the University of Michigan, and a Bachelor’s in History from the University of Chicago.