Volunteer DEF CON hackers dive into America's leaky water infrastructure(theregister.com)
theregister.com
Volunteer DEF CON hackers dive into America's leaky water infrastructure
https://www.theregister.com/2024/11/24/water_defcon_hacker/
26 comments
> What about Department of Homeland Security's Cybersecurity and Infrastructure Security Agency (CISA), where are they in all of this?
The CISA does this type of thing all the time. Here is an example just from their recent news releases in the past week: https://www.cisa.gov/news-events/alerts/2024/11/21/cisa-rele...
The CISA offers to assist infrastructure providers in a number of ways: https://www.cisa.gov/resources-tools/services/assist-visits Presumably the NRWA could have asked the CISA for help with this initiative.
The CISA has 3,000 employees and a budget of $3B, which is a lot, but not enough where they can be involved in everything. For reference there are 12,500 utility scale power plants and 148,000 public water systems in the United States. The scale of the problem means they must be mostly an advisory organization, where most of the work gets done by people at the infrastructure organizations or, sure, volunteers.
Now, I’m not saying that the CISA is doing a good job, I genuinely have no idea. Determining that would take a lot of knowledge, probably insider knowledge, and weighing what they’re doing against what resources they have available. But them not being involved in some random project that showed up in your newsfeed doesn’t mean they have severely failed.
The CISA does this type of thing all the time. Here is an example just from their recent news releases in the past week: https://www.cisa.gov/news-events/alerts/2024/11/21/cisa-rele...
The CISA offers to assist infrastructure providers in a number of ways: https://www.cisa.gov/resources-tools/services/assist-visits Presumably the NRWA could have asked the CISA for help with this initiative.
The CISA has 3,000 employees and a budget of $3B, which is a lot, but not enough where they can be involved in everything. For reference there are 12,500 utility scale power plants and 148,000 public water systems in the United States. The scale of the problem means they must be mostly an advisory organization, where most of the work gets done by people at the infrastructure organizations or, sure, volunteers.
Now, I’m not saying that the CISA is doing a good job, I genuinely have no idea. Determining that would take a lot of knowledge, probably insider knowledge, and weighing what they’re doing against what resources they have available. But them not being involved in some random project that showed up in your newsfeed doesn’t mean they have severely failed.
Agreed, that i also have no idea how CISA is performing; i honestly cannot say anything good or bad - since i have zero data to substantiate my opinuion.
> ...But them not being involved in some random project that showed up in your newsfeed doesn’t mean they have severely failed.
I surely and honestly hope that you are correct!
> ...But them not being involved in some random project that showed up in your newsfeed doesn’t mean they have severely failed.
I surely and honestly hope that you are correct!
Most of our infrastructure was built a long time ago (except roads) and don’t actively see maintenance. No one from the current generations has had to pay for real impactful infrastructure for decades. Now that all the infrastructure is failing or has security holes nobody wants to pay for it to be fixed. If you look up major infrastructure repairs in the last few decades you will see much of what could have been long term fixes were reduced to short term fixes.
Properly fixing infrastructure has become expensive as construction costs have skyrocketed in the last few decades.
Look at the CHIPS bill, huge infrastructure gain but also was a part of the 9% inflation because building this infrastructure is expensive. The American people were warned about it causing inflation but wanted it anyways. When inflation hit they all conveniently forgot the cost of new infrastructure in the US and chose to cling to lies about how disastrous the last 4 years have been.
That is why today no one does anything about problems until a bridge collapses. You can make more money today by passing the buck to the next generation and passing blame to whoever came before you.
Properly fixing infrastructure has become expensive as construction costs have skyrocketed in the last few decades.
Look at the CHIPS bill, huge infrastructure gain but also was a part of the 9% inflation because building this infrastructure is expensive. The American people were warned about it causing inflation but wanted it anyways. When inflation hit they all conveniently forgot the cost of new infrastructure in the US and chose to cling to lies about how disastrous the last 4 years have been.
That is why today no one does anything about problems until a bridge collapses. You can make more money today by passing the buck to the next generation and passing blame to whoever came before you.
CHIPS was signed into law in August 2022. The yearly inflation rate for 2023 was 4.12%. Inflation, especially in the wake of the COVID-19 pandemic, was caused by an extremely large amount of different issues and chalking 9% inflation up to infrastructure spending is a wild jump.
Good thing I didn’t chalk up infrastructure spending as the only cause by stating “part”.
> but also was a part of the 9% inflation
Yes there are other reasons, COVID payments being another reason, which citizens were warned about. The first payment was signed by Donald Trump himself. Any time there is an increase in spending it will cause inflation. Increasing defense spending is another cause which happens yearly.
> but also was a part of the 9% inflation
Yes there are other reasons, COVID payments being another reason, which citizens were warned about. The first payment was signed by Donald Trump himself. Any time there is an increase in spending it will cause inflation. Increasing defense spending is another cause which happens yearly.
Did CHIPS and stimulus checks also cause inflation in Germany, Italy, France, Japan, Canada in Britain? Asking because they all had higher inflation than US.
The pandemic caused global inflation. I don’t know why you’d insinuate increased spending in America directly correlates with increased spending in Europe. I’m not sure what point you’re trying to make either.
Yeah, by no means do i believe that infra.-related topics are easy nor cheap nor fast to resolve...in fact, i believe them to be some of the most diffult and/or complex for myriad reasons...I'm just venting i guess because its more a matter of poor decisions made over time, rather than what is actually and technically possible in our society. But, yeah, passing the buck seems to be what happens, and to our greater detriment.
> Before anyone starts rambling on about politics too much on any side, i blame all sides.
Only one side is consistently obstructionist toward progress.
Only one side is consistently obstructionist toward progress.
I wish we could stop looking for someone to blame or complain about and start looking for solutions. I think that’s the spirit of volunteering here.
Part of looking for solutions requires root cause analysis. Which can be "blameless" but there is at some point a need to figure out where the holes are in a system to be able to patch them. Otherwise people will never know if they're paying for a problem to occur (taxes) and then paying to help fix them too (donations + volunteering)!
Now, this is a good point! I'm actually willing to pay taxes to ensure they are used for the common good. My offspring no longer is school age, but i am STILL willing to pay taxes that pay for elementary school, and such...Because it helps society and the common good in so many ways. But, i think we have to have discussions when those infra.-level things are not providing the benefits because those in power keep making awful decisions...and the next steps don't involve removing said efforts, but rather improving them, bettering their implementations, etc...and that can start with a post-mortem, or RCA, even a simple 5 Whys, whatever it takes to help society stay safe, improve well-being for all, give societal particiapnts a chance at propserity, etc. :-)
I was lamenting the state of society, that's all. I think any volunteers - regardless of area - are true champions. But, i was simplky casting shade to those in power for their failings, and allowing things to trickle to what i believe is a sad state of affairs. I'm just some rando on the web yelling at (the men who "created") the clouds. ;-)
While I see your point, over time, I’ve come to think that cybersecurity is a fundamentally different and indomitable beast. Consider the sheer number of software projects, devices, and products being developed, each inevitably introducing all kinds of bugs, versus the relatively small number of people who truly understand the craft of real offensive security.
The very concept of “offensive security” is indicative of the problem.
If you want to make a secure military base, you do not hire a spec ops team to develop one. If you want to make a bulletproof vest, you do not hire a gunsmith to design new synthetic fibers.
Having offensive teams on hand to verify and validate is necessary, but largely orthogonal to the task of design and development. The skillsets are highly dissimilar.
The fact that people think this is the golden way shows how absolutely intellectually bankrupt the entire commercial cybersecurity industry is on a theoretical level. And the complete inability to protect against the regular and standard threat actors today shows and supports that empirically.
If you want to make a secure military base, you do not hire a spec ops team to develop one. If you want to make a bulletproof vest, you do not hire a gunsmith to design new synthetic fibers.
Having offensive teams on hand to verify and validate is necessary, but largely orthogonal to the task of design and development. The skillsets are highly dissimilar.
The fact that people think this is the golden way shows how absolutely intellectually bankrupt the entire commercial cybersecurity industry is on a theoretical level. And the complete inability to protect against the regular and standard threat actors today shows and supports that empirically.
> If you want to make a secure military base, you do not hire a spec ops team to develop one. If you want to make a bulletproof vest, you do not hire a gunsmith to design new synthetic fibers.
When you can be attacked by groups of 20s something with only a computer and fulltime to attack you via your smartwatch or social networks you would rethink about cybersecurity. Your example is linked to physical but not virtual spaces.
When you can be attacked by groups of 20s something with only a computer and fulltime to attack you via your smartwatch or social networks you would rethink about cybersecurity. Your example is linked to physical but not virtual spaces.
Ah yes: "Don't you understand, the problems in this field are harder. That is why the methodologies that easier fields reject as being grossly inadequate for their easier problems will work on much harder problems." Even ignoring the glaring conceptual holes in that argument, we also have literal decades of objective and empirical evidence that those techniques are still absolute failures when applied to software security.
Furthermore, groups of kids do not constitute a technically sophisticated threat. If you are struggling with them, then your security is grossly inadequate against commonplace and routine threat actors today like individual opportunistic criminals. Which are themselves grossly inferior to commercially motivated cybercrime groups. Which are themselves grossly inferior to state actors who are also commonplace and routine threat actors against national infrastructure.
The fact that commercial IT cybersecurity companies continue to fall prey to such bottom-feeder predators should tell you everything you need to know about the value of their recommendations and "expertise". If you disagree, present a software system developed using "offensive security" focused methodology (which if you present no definition I will define as using offensive specialists to design the security of a software system) that has been explicitly audited to be secure against a small team (let us say 3 person-years) of offensive specialists. And no, the absence of evidence: "There have been no publicly announced hacks on this system" does not constitute the evidence of absence. Only explicit audits that actively test the security of a system provide evidence of absence.
And, just to head-off any claims that is an impossible standard that nobody reaches, I will point to Common Criteria SKPP certifications and the seL4 proofs of correctness as examples of the alternative that has reached such standards.
Furthermore, groups of kids do not constitute a technically sophisticated threat. If you are struggling with them, then your security is grossly inadequate against commonplace and routine threat actors today like individual opportunistic criminals. Which are themselves grossly inferior to commercially motivated cybercrime groups. Which are themselves grossly inferior to state actors who are also commonplace and routine threat actors against national infrastructure.
The fact that commercial IT cybersecurity companies continue to fall prey to such bottom-feeder predators should tell you everything you need to know about the value of their recommendations and "expertise". If you disagree, present a software system developed using "offensive security" focused methodology (which if you present no definition I will define as using offensive specialists to design the security of a software system) that has been explicitly audited to be secure against a small team (let us say 3 person-years) of offensive specialists. And no, the absence of evidence: "There have been no publicly announced hacks on this system" does not constitute the evidence of absence. Only explicit audits that actively test the security of a system provide evidence of absence.
And, just to head-off any claims that is an impossible standard that nobody reaches, I will point to Common Criteria SKPP certifications and the seL4 proofs of correctness as examples of the alternative that has reached such standards.
While i agree that cybersecurity is no way at all an easy thing, i politely disagree that it is indomitable. I'm gonna stretch your intent there to use a cheesy analogy: its like saying humans thought buidling anything over rivers was simply beyond their tech means, so bridges were never invented...But, you know, we have the technology to cross over rivers. (I know, i know, inventing bridges and establishing new standards for safer worlds vis a vis cybersecurity is not the same thing, sure, sure, ok.) :-)
I'm a recently retired software engineer who has been on the board of directors of a small rural mutual water company (i.e., owned by the customers) for ~15 years. We have less than 1,000 customers / shareholders. I'm in my mid-50s and the only one on the board under 75-years old. Our community didn't have wired internet access until eight years ago, and half the community still doesn't have cell reception. You can imagine the level of technical literacy.
It's hard to imagine significant help the government could give us, short of $100K+ / year to hire a security engineer. Even then, how would small utility companies find / evaluate them? We already feel saddled with how many hours of mandatory training we have to go through each year (e.g., board members have to sit through training to remind us not to direct company funds into our own bank accounts - I doubt that's a training issue!). Looking at our two neighboring water companies, their setups have very little in common with ours or each others. So any training would be too generic to be of much use.
I have to give credit to our company's chairman / general manager (himself 84-years old). He works to have diverse expertise on the board. He's a retired wild-land firefighter, plus there's a retired bookkeeper, a retired state employee, a retired farmer, and me. As a group, we have a lot of experience in a lot of areas. I suspect most small, rural water companies haven't found a way to have that breadth of knowledge available. But, even then, there's only so much that two full-time plus two half-time employees can do. And it's hard to have more while keeping rates sane.
On the security side, I'm no expert, but I pay attention. I do my best to help the employees understand the dangers of phishing and of downloading things onto company computers. We have a consumer grade router and none of them know how to get into it to open ports, etc. Even if things were locked down hard, we couldn't afford to hire someone to maintain that state as threats evolve. Our total compensation budget is $150K / year for all employees, and 80% of that is needed for state-licensed water treatment operators.
For us, all of that means being resigned to the fact that hacks will happen. It's more about minimizing the damage and being able to recover. I've pushed for using cloud services for things like customer billing - these services are not the ultimate in security, but they're far better than what we could do with software running on our computers. I've also put append only / offsite / offline backups in place so we can recover from encrypting ransomware (which has hit us before, luckily we could just wipe and re-install / restore).
The biggest thing we've done is around SCADA (software / hardware involved in our water treatment and distribution). When we put SCADA in place ten years ago, I pushed for it to be read-only. That is, it can be used for monitoring and alerting, but not for controlling anything (there's literally no hardware in place to do so). So, hackers can see how much water, chemicals, etc. we're using, but they can't directly change or shutoff the water. They can make us think we need twice the chemicals, but that will be a red flag for the operators who have to manually implement it. Even then, we do manual daily testing of our water in our lab and have monthly testing done by state-licensed independent labs.
But, when we put SCADA in place, I had to stop and think a lot about this given that I wasn't a security expert. And, of course, the vendor was certain it was 100% secure (a red flag in itself). But at least I had experience thinking about software issues / impact. The operators certainly wanted automated control, as that would save them from having to drive to the plant in the middle of the night. And without automated control, we have to use more chemicals, as we can't optimize usage by reacting in real time to changes in pollutants in our source water. But we'd all rather deal with those downsides than find out someone has compromised our water. So it was easy to sway them.
It's far from perfect, but we've mostly limited potential damage to things that aren't deadly, literally or figuratively. Without someone like our chairman / general manager being dedicated to bringing in diverse expertise via the board, I don't see what chance we'd have. And in certain parts of the country, it would be hard to find security (or at least software) expertise to sit on the board.
Something our state is doing to attack the lack of economies of scale (in this and other areas) is trying to force rural water systems to merge. About half in our area have been folded into the water company for the "big" city (5K people), 20-miles away from us. That's too far to move treated water given our local terrain, so they have to keep running the systems from afar. But at least they can spread (e.g., IT) expertise across these systems. Where this has happened near us, water rates have roughly doubled. Maybe that trade-off is okay, at least for those who can afford it?
It's hard to imagine significant help the government could give us, short of $100K+ / year to hire a security engineer. Even then, how would small utility companies find / evaluate them? We already feel saddled with how many hours of mandatory training we have to go through each year (e.g., board members have to sit through training to remind us not to direct company funds into our own bank accounts - I doubt that's a training issue!). Looking at our two neighboring water companies, their setups have very little in common with ours or each others. So any training would be too generic to be of much use.
I have to give credit to our company's chairman / general manager (himself 84-years old). He works to have diverse expertise on the board. He's a retired wild-land firefighter, plus there's a retired bookkeeper, a retired state employee, a retired farmer, and me. As a group, we have a lot of experience in a lot of areas. I suspect most small, rural water companies haven't found a way to have that breadth of knowledge available. But, even then, there's only so much that two full-time plus two half-time employees can do. And it's hard to have more while keeping rates sane.
On the security side, I'm no expert, but I pay attention. I do my best to help the employees understand the dangers of phishing and of downloading things onto company computers. We have a consumer grade router and none of them know how to get into it to open ports, etc. Even if things were locked down hard, we couldn't afford to hire someone to maintain that state as threats evolve. Our total compensation budget is $150K / year for all employees, and 80% of that is needed for state-licensed water treatment operators.
For us, all of that means being resigned to the fact that hacks will happen. It's more about minimizing the damage and being able to recover. I've pushed for using cloud services for things like customer billing - these services are not the ultimate in security, but they're far better than what we could do with software running on our computers. I've also put append only / offsite / offline backups in place so we can recover from encrypting ransomware (which has hit us before, luckily we could just wipe and re-install / restore).
The biggest thing we've done is around SCADA (software / hardware involved in our water treatment and distribution). When we put SCADA in place ten years ago, I pushed for it to be read-only. That is, it can be used for monitoring and alerting, but not for controlling anything (there's literally no hardware in place to do so). So, hackers can see how much water, chemicals, etc. we're using, but they can't directly change or shutoff the water. They can make us think we need twice the chemicals, but that will be a red flag for the operators who have to manually implement it. Even then, we do manual daily testing of our water in our lab and have monthly testing done by state-licensed independent labs.
But, when we put SCADA in place, I had to stop and think a lot about this given that I wasn't a security expert. And, of course, the vendor was certain it was 100% secure (a red flag in itself). But at least I had experience thinking about software issues / impact. The operators certainly wanted automated control, as that would save them from having to drive to the plant in the middle of the night. And without automated control, we have to use more chemicals, as we can't optimize usage by reacting in real time to changes in pollutants in our source water. But we'd all rather deal with those downsides than find out someone has compromised our water. So it was easy to sway them.
It's far from perfect, but we've mostly limited potential damage to things that aren't deadly, literally or figuratively. Without someone like our chairman / general manager being dedicated to bringing in diverse expertise via the board, I don't see what chance we'd have. And in certain parts of the country, it would be hard to find security (or at least software) expertise to sit on the board.
Something our state is doing to attack the lack of economies of scale (in this and other areas) is trying to force rural water systems to merge. About half in our area have been folded into the water company for the "big" city (5K people), 20-miles away from us. That's too far to move treated water given our local terrain, so they have to keep running the systems from afar. But at least they can spread (e.g., IT) expertise across these systems. Where this has happened near us, water rates have roughly doubled. Maybe that trade-off is okay, at least for those who can afford it?
Hey @rented_mule thanks very much for sharing, and i want to thank you for what you and the board and the operators over at the water company do for your community! I'm glad that there are people in the world like you and the rest there, who help the community!
While i understand that funding might be very limited or non-existent, actually, your situation sounds almost exactly the kind that could benefit from support from a government entity. It could run the gamut from free training, to actual technical advisement, support, to direct funding for cybersecurity/protection operations, etc. Tlo be clear, i don't mean that the gov. comes in there and takes over - no, no, no. I mean, that some gov. entity comes to you asks what they can do to help. Sure they can advise on best practice, but its your show, and you should run it your way....its simply that they might have the know-whow and funding to help you and your water company. At least, that's what i figure shoiuld happen to help communities like yours and others! :-)
While i understand that funding might be very limited or non-existent, actually, your situation sounds almost exactly the kind that could benefit from support from a government entity. It could run the gamut from free training, to actual technical advisement, support, to direct funding for cybersecurity/protection operations, etc. Tlo be clear, i don't mean that the gov. comes in there and takes over - no, no, no. I mean, that some gov. entity comes to you asks what they can do to help. Sure they can advise on best practice, but its your show, and you should run it your way....its simply that they might have the know-whow and funding to help you and your water company. At least, that's what i figure shoiuld happen to help communities like yours and others! :-)
>e.g., board members have to sit through training to remind us not to direct company funds into our own bank accounts - I doubt that's a training issue!).
"Have to"? As in 'obligatory?
"Have to"? As in 'obligatory?
Yes, required by state law, every few years for all water company board members. I probably overstated it a bit, but the training is that we can't just grant contracts to ourselves or our own companies without competitive bids, we can't compensate ourselves unreasonably, etc. I'm spending a few hours of my evening to go through that training in a couple of weeks, so I'm feeling a little annoyed about it. :-)
The law was in reaction to a suburban water company board in our state paying each of its members hundreds of thousands of dollars per year until it went bankrupt. That comp is not crazy in massive tech companies, but it is in small water companies. On our board, our compensation is up to $100 off our water bill each month and one bag of trash in the company dumpster each week, if there's room. That's in return for a ~2-hour meeting each month and 2~20 hours of work on various initiatives / committees (or training) each month.
The law was in reaction to a suburban water company board in our state paying each of its members hundreds of thousands of dollars per year until it went bankrupt. That comp is not crazy in massive tech companies, but it is in small water companies. On our board, our compensation is up to $100 off our water bill each month and one bag of trash in the company dumpster each week, if there's room. That's in return for a ~2-hour meeting each month and 2~20 hours of work on various initiatives / committees (or training) each month.
I don't have any inside knowledge to expand on this, but it's interesting that Voting Village has had a public participatory component (website, social media, annual appearance at DEFCON, CFPs guiding messaging on topics on hardware and misinformation); the DARPA AIxCC appears to follow that model, and yet this Franklin Project is much harder to link to a website, social media, mailing list, etc. Is it too early, is it invite-only...?
This seems to be the project's website: http://defconfranklin.com/
Before anyone starts rambling on about politics too much on any side, i blame all sides. What i recall from my basic education so many decades ago is that government should help to provide at least some fundamental areas of infrastructure (e.g. roads and such, etc.), and then commerce (private enterprise, etc.) can take place above it, and than things proceed from there, yada yada.
I don't know what is more basic infrastructure than water and its associated management? Where is the NSA in all of this? What about Department of Homeland Security's Cybersecurity and Infrastructure Security Agency (CISA), where are they in all of this?
Clearly, this story ticked me off, and apologies for that...but, if we're at the stage in society where volunteers is a viable solution for a very fundamental element in life...then clearly lots of other things have severely failed.