Senate passes bill to decrease grid digitization, move toward manual control(utilitydive.com)
utilitydive.com
Senate passes bill to decrease grid digitization, move toward manual control
https://www.utilitydive.com/news/senate-passes-cybersecurity-bill-to-decrease-grid-digitization-move-toward/557959/
188 comments
My intuition from experience is that in sufficiently complex systems (basically all modern hardware + software systems), things break from humans misunderstanding or not knowing the rules far more often than from physical failure.
If there's a reason for using an electronic system, by all means.
But for God's sake air-gap it, compartmentalize functionality with clearly defined interfaces, build simple fault-tolerant monitoring, and make final controls hand-on-metal.
When multi-million dollar warships are crashed because of poor UX, I have zero faith we're safely managing complexity.
If there's a reason for using an electronic system, by all means.
But for God's sake air-gap it, compartmentalize functionality with clearly defined interfaces, build simple fault-tolerant monitoring, and make final controls hand-on-metal.
When multi-million dollar warships are crashed because of poor UX, I have zero faith we're safely managing complexity.
>When multi-million dollar warships are crashed because of poor UX, I have zero faith we're safely managing complexity.
My understanding of the incident is that the auto-detection functions failed and when human functions were not done per SOP due to exhaustion and senior command elsewhere the vessel came into jeopardy. Then when inexperienced and untrained crew members tried too late to evade. An apparently too complex UX was sending ship controls to more than one console. This culminated into one of the multi-million dollar warships crashing.
My understanding of the incident is that the auto-detection functions failed and when human functions were not done per SOP due to exhaustion and senior command elsewhere the vessel came into jeopardy. Then when inexperienced and untrained crew members tried too late to evade. An apparently too complex UX was sending ship controls to more than one console. This culminated into one of the multi-million dollar warships crashing.
A lot of problems with automation vs manual controls is that the line between those tends to be blurry.
It's not that automation wouldn't work in the absolute majority of cases, and that we couldn't define the cases where automation will have to know it cannot make it.
It's not that humans would always be better than automation: humans make mundane mistakes and casual accidents happens, and humans make bad judgements also in extreme situations.
Looking at the catastrophes in aviation and shipping from the recent years the problem seems to lie where automation and human controls interact.
In the typical case the automation is not completely switched off while the human still tries to take control of the system and some override functionality still triggers the automation without the human realising that.
Maybe there's a sensor failure that causes the automation to misbehave. Then the human controller turns off the autopilot and begins to apply the proper fix but the electronic system, through its faulty sensors, sees it differently and concludes that the human is doing something way too dangerous and doesn't let him. Then a tug of war begins with the human controller where the automation is thinking that the situation is dire and the ship must be saved while the human realises that something is not working properly but in the heat of the moment doesn't have time to diagnose or narrow down to shutting off automation and just keeps applying more.
If only the automation knew it can't "see" properly and the human is already trying to fix things, or if only the human knew part of the automation is still active and secretly, with no indication, countering the proper corrections by the human controller.
It's not that automation wouldn't work in the absolute majority of cases, and that we couldn't define the cases where automation will have to know it cannot make it.
It's not that humans would always be better than automation: humans make mundane mistakes and casual accidents happens, and humans make bad judgements also in extreme situations.
Looking at the catastrophes in aviation and shipping from the recent years the problem seems to lie where automation and human controls interact.
In the typical case the automation is not completely switched off while the human still tries to take control of the system and some override functionality still triggers the automation without the human realising that.
Maybe there's a sensor failure that causes the automation to misbehave. Then the human controller turns off the autopilot and begins to apply the proper fix but the electronic system, through its faulty sensors, sees it differently and concludes that the human is doing something way too dangerous and doesn't let him. Then a tug of war begins with the human controller where the automation is thinking that the situation is dire and the ship must be saved while the human realises that something is not working properly but in the heat of the moment doesn't have time to diagnose or narrow down to shutting off automation and just keeps applying more.
If only the automation knew it can't "see" properly and the human is already trying to fix things, or if only the human knew part of the automation is still active and secretly, with no indication, countering the proper corrections by the human controller.
Operators may cause slight SOP deviations in emergency situations. Anticipating this in design between human controls and automated function is going to be very hard. I hope we can get voice input or other advanced input devices setup for communication between systems.
The input isn't the problem IMHO, it's the output.
If a human is in the loop at all then it's because they may be expected to make a decision. And that decision is based on information. And that information needs to be presented unambiguously.
One of the smartest things I read on feedback here was to start with a balance-in, balance-out counting system.
It's unambiguous: if the number of things being processed and output is different than the number of things input, that tells me something.
And that's where automated systems get dangerous. It's incredibly risky if instead of informing and shrinking their response space, they expand it in an attempt to auto-correct the issue.
If a human is in the loop at all then it's because they may be expected to make a decision. And that decision is based on information. And that information needs to be presented unambiguously.
One of the smartest things I read on feedback here was to start with a balance-in, balance-out counting system.
It's unambiguous: if the number of things being processed and output is different than the number of things input, that tells me something.
And that's where automated systems get dangerous. It's incredibly risky if instead of informing and shrinking their response space, they expand it in an attempt to auto-correct the issue.
[deleted]
Sounds like these systems need a little red killswitch under a plexiglass cover. It's baffling to me that anyone who's ever designed an automated system thinks there won't be situations where a human has more complete awareness of what's happening and needs to unambiguously wrest all control.
It's usually not that simple. The majority of complex systems have substantial portions of their operation effectively invisible to us.
Electricity, radiation, remote or sealed mechanical systems, deeply layered software.
In all of these, you need systems to communicate actual state to a human to effect even a manual resolution.
Electricity, radiation, remote or sealed mechanical systems, deeply layered software.
In all of these, you need systems to communicate actual state to a human to effect even a manual resolution.
I wonder what would happen in an actual battle with the fog of war when people are pushed beyond exhaustion. What happens when senior command is dead or missing?
There exists a very distinct line between routine/exercise and war.
You may have missed the opportunity to experience this distinction first hand, but I can assure you with the utmost confidence, the meta does indeed change.
You may have missed the opportunity to experience this distinction first hand, but I can assure you with the utmost confidence, the meta does indeed change.
Though one could say that this was due to too little automation rather than too much -- the default behavior could be for the ship to steer itself away from another vessel or shallow water unless overridden (setting off alarms that this action is being taken)
My $30K car does a fair job of nudging the steering wheel if it thinks I'm running off the road, surely a $2B battleship could have some basic collision avoidance capability too.
My $30K car does a fair job of nudging the steering wheel if it thinks I'm running off the road, surely a $2B battleship could have some basic collision avoidance capability too.
Unfortunately military ships are expensive and have strict schedules. You may be able to drive your car to a shop any day to diagnose a sensor error but military vessels dont have that luxury.
To my knowledge the detection systems (for detecting ships ect) were either not calibrated/failed making a ship impossible to detect. There may have been a serious lack of training spotting sensor failure too. The ship was due for repairs and had been put off. Also, according to SOP there should have been someone making rounds and looking out and identifying ships. Because honestly how do you miss a huge cargo ship lit up like a Christmas tree
There were multiple critical failures leading to the crash. Bad training, poor maintenance, and lax procedural compliance.
To my knowledge the detection systems (for detecting ships ect) were either not calibrated/failed making a ship impossible to detect. There may have been a serious lack of training spotting sensor failure too. The ship was due for repairs and had been put off. Also, according to SOP there should have been someone making rounds and looking out and identifying ships. Because honestly how do you miss a huge cargo ship lit up like a Christmas tree
There were multiple critical failures leading to the crash. Bad training, poor maintenance, and lax procedural compliance.
The problem with automation is that it works perfectly until it runs into that edge case which nobody thought about. The problem with the real world is that there are infinity edge cases.
I didn't mean for the automation to be primary - I meant for it to be a llast-ditch collision avoidance system, like the steering wheel nudge on my car.
Humans should still pilot the ship, but if they are going to let a collision happen, then the ship should try to avoid it even without human intervention (unless they press the "I know what I'm doing and want to do it anyway" button after the ship sounds the alarm saying it's taking evasive action)
So the system doesn't have to handle all of the edge cases, just the "We're on a collision course with that ship ahead" case.
Humans should still pilot the ship, but if they are going to let a collision happen, then the ship should try to avoid it even without human intervention (unless they press the "I know what I'm doing and want to do it anyway" button after the ship sounds the alarm saying it's taking evasive action)
So the system doesn't have to handle all of the edge cases, just the "We're on a collision course with that ship ahead" case.
All of which 95% of those edge cases are most likely programmed leading to saving humans countless times in the most situations to the point they don't even realize it.
Why do you think flying is so safe? It's because of automation and redundant systems.
Do things go wrong? Yes. But imagine how many things would go wrong if you gave people a 15,000 line check list.
Why do you think flying is so safe? It's because of automation and redundant systems.
Do things go wrong? Yes. But imagine how many things would go wrong if you gave people a 15,000 line check list.
I'd like to use this in my .signature.
> there are infinity edge cases.
this is simply not true. there are finite particles in discrete space so there's no infinity here.
this is simply not true. there are finite particles in discrete space so there's no infinity here.
I would call it O(N!) with N increasing over time opposed to infinite and essentially something everything technically faces automated or not - it is just those lacking agency cannot change deliberately.
Not because it isn't effectively beyond linear counting or coverage but because it describes how it grows - in different permutations. Especially with actual other actors involved and literal arms races. Sure your expensive plate armor may stop blades, arrows, and even current musket fire but would it stop cannonfire?
Not because it isn't effectively beyond linear counting or coverage but because it describes how it grows - in different permutations. Especially with actual other actors involved and literal arms races. Sure your expensive plate armor may stop blades, arrows, and even current musket fire but would it stop cannonfire?
What event was this?
Based on google results, it appears to be an incident where the USS McCain collided with the Alnic MC near Singapore. She was getting off course, and poor UI made it so hard to steer her back that she crashed. Injuries and deaths resulted.
https://medium.muz.li/ui-confusion-almost-sunk-a-us-destroye...
https://medium.muz.li/ui-confusion-almost-sunk-a-us-destroye...
Sounds like the US Navy destroyer that collided [0] with a cargo ship.
[0] https://en.wikipedia.org/wiki/USS_Fitzgerald_and_MV_ACX_Crys...
[0] https://en.wikipedia.org/wiki/USS_Fitzgerald_and_MV_ACX_Crys...
More than one large power outage has been caused by a human operater manually grounding live transmission lines by mistake.
Sounds like an interesting situation, got any good links on the topic?
It's an extremely common situation with interesting solutions. If you've experienced the power going out, then coming on a few seconds later, then going out, and coming back on a few seconds later, then going out and staying out you've experienced a "recloser" attempting to clear a power fault (tree falling on power line perhaps, the recloser goes through the motions in an attempt to burn the thing away and clear the fault).
A quick google turned up this fairly lengthy page https://www.electricaltechnology.org/2018/03/electrical-prot... but the protective systems are covered in depth in basic power transmission tech texts. Also here https://en.wikipedia.org/wiki/Recloser
A quick google turned up this fairly lengthy page https://www.electricaltechnology.org/2018/03/electrical-prot... but the protective systems are covered in depth in basic power transmission tech texts. Also here https://en.wikipedia.org/wiki/Recloser
Cool, thanks!
Nearly this exact scenario has been bugging me for a couple months. I started thinking about it due to deployment issues.
Why don't we have a proverbial two key system for certain activities like deploying to production or deleting user accounts, or backups?
We make do with things like pull requests, but everywhere we've ever had retros, I've run into situations where someone did something very stupid, somebody else rubber stamped it, and now we have a mess. Approvals on an action do not approximate the ritual sobriety that is embodied in two independent humans deciding to turn a key.
There are some people for whom these two actions can be considered equivalent, but we tend to bag on them for their meticulousness.
I think I'm wanting a tool where I type "deploy 1.0.12345 to production" and it sits there waiting until someone else types "deploy 1.0.12354 to production", and then it stops us because one of us transposed some digits.
Why don't we have a proverbial two key system for certain activities like deploying to production or deleting user accounts, or backups?
We make do with things like pull requests, but everywhere we've ever had retros, I've run into situations where someone did something very stupid, somebody else rubber stamped it, and now we have a mess. Approvals on an action do not approximate the ritual sobriety that is embodied in two independent humans deciding to turn a key.
There are some people for whom these two actions can be considered equivalent, but we tend to bag on them for their meticulousness.
I think I'm wanting a tool where I type "deploy 1.0.12345 to production" and it sits there waiting until someone else types "deploy 1.0.12354 to production", and then it stops us because one of us transposed some digits.
we have some of these systems at AWS where you need two people to approve a commit, or a type of change operation. Basically, introduce friction at various points in the name of safety.
Yeah I specifically called this out as a stopgap not a solution. For reasons of ergonomics and suggestibility (see also the primacy effect) you are more likely to make a bad judgement call when prompted into it than if you have to reason it out for yourself.
Someone did release a Pam module? That required confirmation before someone sudo-ed. I thought it was Stripe but I can't find it right now.
Hey, that was me! See the sister comment.
It's actually a plugin for sudo (surprisingly yes, sudo has plugin capabilities[1]) and not PAM. I had originally developed it as a PAM module, but the sudo plugin API allows for the neat trick where the TTY is mirrored.
[1] https://linux.die.net/man/8/sudo_plugin
It's actually a plugin for sudo (surprisingly yes, sudo has plugin capabilities[1]) and not PAM. I had originally developed it as a PAM module, but the sudo plugin API allows for the neat trick where the TTY is mirrored.
[1] https://linux.die.net/man/8/sudo_plugin
[deleted]
We didn’t “release” it, but I worked on a dual-control PAM module: https://github.com/cjdev/dual-control
It's not necessarily for deploys (though you certainly could make a workflow that uses it to do so!) but I developed a dual-control plugin for sudo while at Square. We use it extensively.
https://github.com/square/sudo_pair
In action: https://raw.githubusercontent.com/square/sudo_pair/master/de...
https://github.com/square/sudo_pair
In action: https://raw.githubusercontent.com/square/sudo_pair/master/de...
Doesn’t that encourage using sudo in interactive mode rather than elevating for just the required commands?
> Doesn’t that encourage using sudo in interactive mode rather than elevating for just the required commands?
It might, but policy could specify not allowing interactive mode for situations that dictate a two-man rule. Violation of said policy would necessitate a review and possible HR-related action depending on how serious the company wants to treat things.
It might, but policy could specify not allowing interactive mode for situations that dictate a two-man rule. Violation of said policy would necessitate a review and possible HR-related action depending on how serious the company wants to treat things.
I suppose you're right, but the reason that's generally advised against is so people don't run stupid stuff without thinking while root. I'd guess you're less likely to do that if a peer is watching.
And to stouset: that's a seriously cool project!
And to stouset: that's a seriously cool project!
I have seen a contactor SCADA controller (high voltage physical relay) for a high voltage transmission line at a Florida substation on the public internet (no airgap, no vpn) during an audit.
Apply brakes while we work on the steering. This is good policy. “Internet Of Shit” is no joke.
Apply brakes while we work on the steering. This is good policy. “Internet Of Shit” is no joke.
In a hacking group I'm part of, we had an address that ended up being a passwordless VNC to what looked like a centrifuge controller. It was at a place whose IP tracked to CERN.
We reported it as fast as we fucking could. It took about 6 hrs for them to get it off the public internet with no password.
We reported it as fast as we fucking could. It took about 6 hrs for them to get it off the public internet with no password.
Thank you for reporting this!
We do what we can.
From our assessment, it looks like that was a complete mistake. And those do indeed happen. But what if someone were to have found it and were malicious, and started randomly mashing buttons?
That's why our jobs as sysads are never done. We occasionally do boneheaded stuff. Users also do. And so do contractors. And I'll do what I can when I see something wrong. Professional courtesy.
From our assessment, it looks like that was a complete mistake. And those do indeed happen. But what if someone were to have found it and were malicious, and started randomly mashing buttons?
That's why our jobs as sysads are never done. We occasionally do boneheaded stuff. Users also do. And so do contractors. And I'll do what I can when I see something wrong. Professional courtesy.
Industrial controls at least work most of the time. Can't quite say that about consumer gear.
That does not surprise me in the least. Some of the stuff I came across doing DD would make your hair stand on end, after 130 companies I'm more surprised by the ones that do it right than the ones that have no clue.
Agree entirely.
Why not create a closed government only internet or intranet where all grids connect to each other but none have access to the main internet. No computer in the network can connect to the internet and the govnet, maybe even have a different protocol and other unique features. They could also use ai to determine if a command to say shut something important off makes sense if not notify someone in charge before following through on suspect actions.
Logistics and attendant expenses most likely - especially with practice to gain any actual benefits.
Say everyone has a registered nonrepudiatable cryotographic certificate - a decent idea in theory. However apply it to every accessible grid communication part including ones which may cycle in and out and could have secrets extracted. Suddenly it doesn't provide as much benefit if a hacked SmartMeter can get in anyway and there is deniability. Better than nothing but you could have likely gotten better bang for your buck with strict firewall rules or hiring a swarm of consultants.
Similarly vetting commands has two problems - one is that reaction time may be critical. Second the judgements themselves are complexity and may be a source for errors - especially when they are "triage" ones that cause a shutdown or damage to avoid greater damage
Say everyone has a registered nonrepudiatable cryotographic certificate - a decent idea in theory. However apply it to every accessible grid communication part including ones which may cycle in and out and could have secrets extracted. Suddenly it doesn't provide as much benefit if a hacked SmartMeter can get in anyway and there is deniability. Better than nothing but you could have likely gotten better bang for your buck with strict firewall rules or hiring a swarm of consultants.
Similarly vetting commands has two problems - one is that reaction time may be critical. Second the judgements themselves are complexity and may be a source for errors - especially when they are "triage" ones that cause a shutdown or damage to avoid greater damage
> Why not create a closed government only internet or intranet where all grids connect to each other but none have access to the main internet.
An air gap didn't help Iran against Stuxnet, or the US DoD against agent.btz:
* https://en.wikipedia.org/wiki/2008_cyberattack_on_United_Sta...
It wouldn't hurt, and it's probably worth doing anyway as a risk mitigation, but keeping it completely sanitary would be impossible.
An air gap didn't help Iran against Stuxnet, or the US DoD against agent.btz:
* https://en.wikipedia.org/wiki/2008_cyberattack_on_United_Sta...
It wouldn't hurt, and it's probably worth doing anyway as a risk mitigation, but keeping it completely sanitary would be impossible.
The “smart grid” we are working towards does things like scheduling your AC and dishwasher to turn on/off depending on supply and demand fluctuations. To that end, every node, both consumers and energy producers, needs to be connected. To do this completely independent from the existing internet would amount to doubling the cost of our communication infrastructure. It would also defeat your purpose, because suddenly everyone on the grid has access to the network again, and could try to attack it.
You also need some connection, somewhere, to the “regular” internet. As just one example, consumption data is the basis for billing, which needs to connect to financial infrastructure.
And, finally, even air-gapped systems are vulnerable. See Stuxnet.
You also need some connection, somewhere, to the “regular” internet. As just one example, consumption data is the basis for billing, which needs to connect to financial infrastructure.
And, finally, even air-gapped systems are vulnerable. See Stuxnet.
Does smart-grid tech work any better than smart-home tech? So far most 'smart' tech hasn't had a good track record on security or reliability, certainly not at the level you would trust your major infrastructure to it.
Keep in mind most "smart home" technology is marketed at consumers who have no idea what SSL/TLS is (nore do many care sadly even though they should), and are meant to be as cheap as possible.
While I have no clue what the security or reliability looks like in industrial applications, I would hope they evaluate their solutions with reliability in mind.
While I have no clue what the security or reliability looks like in industrial applications, I would hope they evaluate their solutions with reliability in mind.
>While I have no clue what the security or reliability looks like in industrial applications, I would hope they evaluate their solutions with reliability in mind.
Keep hoping... The number of industrial systems deployed that are connected to the internet with the default passwords unchanged or no passwords is nauseating.
https://threatpost.com/exposing-scada-systems-shodan-110910/...
Keep hoping... The number of industrial systems deployed that are connected to the internet with the default passwords unchanged or no passwords is nauseating.
https://threatpost.com/exposing-scada-systems-shodan-110910/...
Serious question. I've been hearing about industrial applications/medical etc. having default passwords exposed on the internet for years.
How have they not been exploited/fixed already? Shodan's made it clear for years, of course anybody can use zmap/masscan.
How have they not been exploited/fixed already? Shodan's made it clear for years, of course anybody can use zmap/masscan.
People that know the passwords are generally ethical. Plus the risk reward isn't there. Say you have a hack for a pacemaker and you kill 1000 people. Then what? Literally every nation on Earth will be looking for you and you will either rot in prison for life or be put to death. And if the wrong country gets you first there will likely be a lot of torture along the way.
Without having hacked a database of patient to device mappings, it's not like you can ask for ransom.
Without having hacked a database of patient to device mappings, it's not like you can ask for ransom.
Frame someone you don't like, short the vendor's stock, or separately hack a hospital to get the patient records. You don't even have to kill them for that, just a recordable, scary event.
Also, you may already want someone in particular dead and are just looking for a vector. This seems almost untraceable.
Also, you may already want someone in particular dead and are just looking for a vector. This seems almost untraceable.
I've seen several talks where they demoed showing access to hydroelectric dams, etc.
I'm thankfully not this person, but at a basic level I could see some bored teen turning off a dam just for fun.
Honestly the only reason I don't is because it would be "wrong" to do, but the internet is full of people, it's certainly not a stretch that somebody would want to.
I'm thankfully not this person, but at a basic level I could see some bored teen turning off a dam just for fun.
Honestly the only reason I don't is because it would be "wrong" to do, but the internet is full of people, it's certainly not a stretch that somebody would want to.
It can happen by accident. Some teen has a bot that scans around bricking devices with weak security; bricks critical infrastructure.
Most industrial application also done know or care what ssl/TLS is (however this is starting to change). I have had industrial control systems reboot by just doing an nmap scan.
This
It's barely better than iot stuff. And some "older" systems probably still need NT4 to run (I mean, if you're lucky)
It's barely better than iot stuff. And some "older" systems probably still need NT4 to run (I mean, if you're lucky)
Hah. My hope died with the 30 ICS-CERT emails I’m bombarded with each day.
It is just because ICS/IoT vendors do not care/know a thing about cybersecurity. If you push security development practices, it should really improve the situation. Just prevent an access to the market until they pass the red team audit. Would surely motivate them enough.
For the utilities, absolutely. The utilities have hard requirements of delivering power to all customers while maintaining grid and frequency stability, so the ability to shape demand and supply quickly is at least one very valuable knob.
Yes. The modern sensors and controls being built in to US electric transmission and distribution systems have resulted in a far better run grid: more stable, more reliable, more efficient, fairer, and enabling things like real-time market making which enables rapid improvement using new technologies such as grid storage batteries, etc.
It does come with some vulnerabilities. I'm not sure what the best way forward on those is; I suspect this bill's effects, if any, will not result in a more resilient or less vulnerable grid system. Attackers will just find different leverage points to attack.
It does come with some vulnerabilities. I'm not sure what the best way forward on those is; I suspect this bill's effects, if any, will not result in a more resilient or less vulnerable grid system. Attackers will just find different leverage points to attack.
Somewhat related anecdote:
I live in the Finnish countryside, and since much of the power grid out here is still ancient, there are power outages rather often.
However, if the power goes out, there's always something that tries to power everything back on automatically after exactly 30, 60, 120 and 240 seconds. If something's still wrong, the power will simply flick on and then off within a few seconds. If an outage lasts more than four minutes, it usually takes much longer to get it running again (I assume it's because it then triggers an alert that will be handled by a human operator).
I would assume there's a lot of similar stuff going on in the US power grid, considering it's immense size what I suspect is a somewhat similar mix of new and very old equipment.
And yes, surge protectors are mandatory here.
I live in the Finnish countryside, and since much of the power grid out here is still ancient, there are power outages rather often.
However, if the power goes out, there's always something that tries to power everything back on automatically after exactly 30, 60, 120 and 240 seconds. If something's still wrong, the power will simply flick on and then off within a few seconds. If an outage lasts more than four minutes, it usually takes much longer to get it running again (I assume it's because it then triggers an alert that will be handled by a human operator).
I would assume there's a lot of similar stuff going on in the US power grid, considering it's immense size what I suspect is a somewhat similar mix of new and very old equipment.
And yes, surge protectors are mandatory here.
> If an outage lasts more than four minutes, it usually takes much longer to get it running again (I assume it's because it then triggers an alert that will be handled by a human operator).
I read a statistic at some point (which I can no longer find the reference for) that power outages tend to be bimodal: either they last a short time (e.g. <5min) or they last a long time. There's hardly anything in between.
In the US, utilities report outages longer than five minutes:
* https://www.eia.gov/todayinenergy/detail.php?id=35652
I read a statistic at some point (which I can no longer find the reference for) that power outages tend to be bimodal: either they last a short time (e.g. <5min) or they last a long time. There's hardly anything in between.
In the US, utilities report outages longer than five minutes:
* https://www.eia.gov/todayinenergy/detail.php?id=35652
> there's always something that tries to power everything back on automatically after exactly 30, 60, 120 and 240 seconds
Sounds like a recloser, which tries a number of times to restore power under the assumption that most faults are transient (for instance, a branch falling in the wires and then sliding to the ground). After a set number of retries, it gives up and stays open; at that point, a human has to locate the fault, clear it (for instance, poking the fallen branch with a stick until it falls to the ground), and then tell the recloser to close the connection again.
Sounds like a recloser, which tries a number of times to restore power under the assumption that most faults are transient (for instance, a branch falling in the wires and then sliding to the ground). After a set number of retries, it gives up and stays open; at that point, a human has to locate the fault, clear it (for instance, poking the fallen branch with a stick until it falls to the ground), and then tell the recloser to close the connection again.
Yes, that sounds to be it. I can imagine such systems takes care of quite a lot of incidents in large power grids.
It probably will yield good results for the next decade but will set a bad precedent for the world in general and back us into a corner as confidence in the security for this stuff improves and the next decades passes.
Software does not appear to be on a trajectory of increased security or reliabilty. It has always been hard to verify correctness, let alone measure how vulnerable and easy to exploit software is. Testing software has not been a suitable replacement to a large technical user base constantly tweaking and improving said software.
If software were to become reliable, I'm sure those pioneers in reliable software can pave the way to reasonable use in mission critical, high value espionage targets.
Until then, legislation like this will pave the way back to what has worked for centuries.
If software were to become reliable, I'm sure those pioneers in reliable software can pave the way to reasonable use in mission critical, high value espionage targets.
Until then, legislation like this will pave the way back to what has worked for centuries.
This is an important step in safe guarding our collective experience of power plants.
Consider that the way we currently interact with power plants: "Someone needs to stay behind and manually make the boilers explode! I'll do it! Tell Laura I love her." pales in drama to "OK, let's see, Power / Input / Boiler / Max, confirm, confirm, confirm. OK, let's run!"
Consider that the way we currently interact with power plants: "Someone needs to stay behind and manually make the boilers explode! I'll do it! Tell Laura I love her." pales in drama to "OK, let's see, Power / Input / Boiler / Max, confirm, confirm, confirm. OK, let's run!"
This reminds me of the 2004 Battlestar Galactica reboot. All of the ships in the fleet are modern and get hacked and taken over by an outside force. The "Galactica" is being decommissioned because it's old and it's systems are linked in with the rest of the fleet. Hence it is able to withstand the attack and save humanity.
It is quite alarming if we need to move away from digital connected tech for electrical grid monitoring and management.
What about banks? What about hospitals? storage facilities? record keeping? These are the core underpinnings of a functioning society. We are already using digital-only systems for these. If these can be disrupted or corrupted, then society will come falling apart quickly. If we truly cannot secure a digital system for electrical grid and hence we should move away from digital, then we should be doing the same for these other critical systems as well. Is this even practical now? Shouldn't we be going down the path of inventing chips, systems, algorithms, protocols and proof methods to secure these systems?
There's probably an ideal system down the route of more high tech, but in the real world they would never be implemented perfectly. As complexity increases so does room for security issues.
I don't see the problem with striking a balance with technology and putting in low-tech solutions for security reasons. So yeah I think many of those systems should implement similar measures.
I don't see the problem with striking a balance with technology and putting in low-tech solutions for security reasons. So yeah I think many of those systems should implement similar measures.
We are capable of protecting our assets. We just have an administration that prefers to sacrifice efficiency to lock in a dying energy industry.
Isn't this great success of 'Russia/China' so-called hacking ?
There’s a reason our nuclear system still operates on floppy disks.
Floppy-disk based systems are not the only way to ensure security/reliability, but it's far better than a half-baked system that isn't properly audited. Systems we rely on must be dead simple and reliable.
There was no alternative to floppy disks when they were built though. Was there a deliberate choice to use an older technology?
I'd assume that's more about not fixing something that isn't broken. (Because if you break it, you might accidentally annihilate humanity!)
Do you have a source on this?
Google it, it seems to be a generally known fact: https://www.cnn.com/2016/05/26/us/pentagon-floppy-disks-nucl...
https://www.cnn.com/2016/05/26/us/pentagon-floppy-disks-nucl...
It was a popular tech story a few years ago.
It was a popular tech story a few years ago.
This is when I head off to n-gate.com
Why not simply stop networking these systems together and/or connecting them to the internet?
There would be no way to do reporting on any of the devices in the field without some form of network connection. This delays response time if, for example, a recloser failed to operate, nobody would be aware unless someone physically checked the device. With networked SCADA systems, this becomes an immediate notification to the control room.
I'm a fan of manual systems, but within reason though. The point is, you're never going to be 100% protected. People can still break into substations and compromise the grid rather easily.
I'm a fan of manual systems, but within reason though. The point is, you're never going to be 100% protected. People can still break into substations and compromise the grid rather easily.
Read only monitoring is fine. “Write” ability must be heavily permissioned, audited, and governed. Failing that, no physical “write” access.
[deleted]
Lets take my example and say you had a recloser that is stuck, but could be fixed by sending a command that would perform a manual close. Or maybe, you wanted to set the time threshold for the closing.
a) Would you send a lineman out to the recloser and have them manually configure it, spending probably hours doing so?
b) Or would you have some operator press a button in a piece of software to engage the recloser (thus, ultimately, writing whatever register in the device to perform the closing)?
Now multiply that scenario to the potentially hundreds/thousands of reclosers in the field.
Companies are going to choose b) every time.
a) Would you send a lineman out to the recloser and have them manually configure it, spending probably hours doing so?
b) Or would you have some operator press a button in a piece of software to engage the recloser (thus, ultimately, writing whatever register in the device to perform the closing)?
Now multiply that scenario to the potentially hundreds/thousands of reclosers in the field.
Companies are going to choose b) every time.
Yes, I agree, which is why regulation is crucial to ensure a balance is struck between operational efficiency and security.
If you want to interrogate recloser behavior remotely (real time and for historical logging), entirely reasonable. If you want to reclose manually or update trip and reclose thresholds, your commands or setting updates must be authenticated, logged, and cryptographically signed. If the ACR encounters lockout, you roll a truck (which you're probably going to do regardless, as lockout indicates a non-transient fault).
If you want to interrogate recloser behavior remotely (real time and for historical logging), entirely reasonable. If you want to reclose manually or update trip and reclose thresholds, your commands or setting updates must be authenticated, logged, and cryptographically signed. If the ACR encounters lockout, you roll a truck (which you're probably going to do regardless, as lockout indicates a non-transient fault).
Alright, your response is a lot clearer now. However, this is how it currently is with NERC CIP, so I fail to see how having legislation that facilitates more manual devices really solves anything.
Why are you reading something if you're not going to take action based on that data? Reads are de-facto writes elsewhere, so they need to be authenticated.
Action can be taken with analog, non networked controls.
If reads or communications can be spoofed, operators can be convinced to take the wrong actions with their analogue controls.
The "analogue control" may be some dude rolling up in a truck and doing the work. The "analogue control" is operated via radio or a ticketing dispatch system.
[deleted]
Integrity of the monitoring apparatus isn’t out of scope, but simply too long of a discussion to be captured in this thread (vs “button that makes it break set to public”).
possibly, but what do you consider “write ability”? someone mentioned in another comment that if you control enough high power devices that report status to the grid (eg car chargers) so that the grid can take action (so read only but some other system uses it to take action) you can do plenty of damage
they don’t really have write access, but they kinda do?
they don’t really have write access, but they kinda do?
Write access, in my opinion, is where you have enough control to remotely override life critical or physical control systems that could cause death and/or significant physical damage to infrastructure and related failsafe systems can be bypassed or their thresholds exceeded intentionally.
An Internet rando should not have the ability to disconnect your 700kva interconnect over the public internet. If you breach Tesla and can command enough vehicles to charge in a constrained area to overload local distributed infrastructure, failsafes should kick in and physically segregate loads from the local grid.
Disclaimer: This is only my opinion as an infosec practitioner having done some infosec/GRC consulting for investor owed and coop utilities for FERC compliance.
An Internet rando should not have the ability to disconnect your 700kva interconnect over the public internet. If you breach Tesla and can command enough vehicles to charge in a constrained area to overload local distributed infrastructure, failsafes should kick in and physically segregate loads from the local grid.
Disclaimer: This is only my opinion as an infosec practitioner having done some infosec/GRC consulting for investor owed and coop utilities for FERC compliance.
This is exactly what is done in practice. Critical systems (i.e. control room software and hardware) are air gapped and operate in a DMZ.
The only real vector one has into such a system (assuming you can't physically access it) is by attacking the 'inputs' to the SCADA network.
Attack vectors like stuxnet or petya require physical access to these systems (i.e. a usb key getting plugged into an operation terminal).
The only real vector one has into such a system (assuming you can't physically access it) is by attacking the 'inputs' to the SCADA network.
Attack vectors like stuxnet or petya require physical access to these systems (i.e. a usb key getting plugged into an operation terminal).
But... airgapping doesn't stop damages from Aurora type of attacks. So even with top-level grid controls airgapped, if the attacker is at the switch for enough vulnerable lower-level consumers and can synchronize enough switches (car chargers, solar inverters, local substations, etc) the potential damages can still be very significant. As we saw in the Ukraine power grid attacks, you don't even need to go after critical systems. Quantity has its own quality, as the saying goes.
> This is exactly what is done in practice.
I think you’ve got this backwards. This is exactly what is done _in theory_. In practice many of these systems are not. When I was working as a consultant, I saw so many exposed SCADA systems.
I think you’ve got this backwards. This is exactly what is done _in theory_. In practice many of these systems are not. When I was working as a consultant, I saw so many exposed SCADA systems.
Typical SCADA system from the eighties will have an open port listening to UDP traffic, if you're really lucky without any kind of checksum where each bit in a packet will toggle a relay and reply with the state of all its input channels in an ACK packet. Given the state of these systems it is amazing that they are not compromised spectacularly every other day. Or maybe they are but it does not make the news. Plenty of this stuff is running on BASIC stamps and such (no kidding) used to network systems that are even older.
There are also NERC compliance requirements that require USB ports be disabled at and OS or BIOS level.
Maybe there are still benefits to have a network, but not using internet. Couldn't we build a network with a subset of internet features but which are all formally proved, and let humans take care of the rest ?
Internet is fine as a network, it is resilient to damage. What is sufficient to make it secure is to restrict the network features of the facility server software to one command, SHOW_STATUS. Control of operation over network should be impossible.
My spidey sense tingles every time someone uses the S word in regards to system design or security.
A system which "should" only allow read access is a system that will most likely have flaws which allow for far more, especially given the governments tendency to underspend on salaries for the people responsible for implementation.
A system which "should" only allow read access is a system that will most likely have flaws which allow for far more, especially given the governments tendency to underspend on salaries for the people responsible for implementation.
Random damage? Yes, quite resilient. Targeted attack? Not so much.
It doesn't stop attacks.
Stuxnet successfully attacked air gapped systems.
Stuxnet successfully attacked air gapped systems.
It does stop attacks. It just doesn’t stop all attacks. Just because a security measure, by itself, isn’t perfect doesn’t mean it isn’t worth implementing.
Arguably that was a failure of physical or personnel controls. Someone was able to take media in. A physical device was allowed to be connected.
Still makes them exponentially harder since you need to get someone to connect an infected device to the internal network somehow.
IIRC, stuxnet successfully targeted and exploited weak humans. The air gap didn't fail, an idiot worker with a USB drive was the vector.
Good to see some more "common-sense" in contrast to the "Internet everything" trend that's taken hold in some areas of the industry. Some of the control systems in the national grid have been in use for over a century, and are likely to continue working indefinitely if kept maintained.
We need smart-gridness to reduce our dependency on idle fossil-burning power plants as a safety net to balance the now much less predictable grid. Strictly ecologically speaking, this seems problematic.
This is an important point.
Increasing renewable penetration into a power system leads to:
-> reduced inertia
-> increasing Rate of Change of Frequency (RoCoF)
-> reduced time to react
If you rely entirely on manual systems to react to frequency changes on a grid with high renewable energy penetration (e.g. the Irish grid), you'd really struggle to avoid daily blackouts without decommissioning a bunch of wind farms.
It is by using such automated control schemes that countries like Ireland are able to push the limit on wind power penetration.
If I was being truly cynical, I would think that a push towards mandating manual control (rather than mandating that it is available as a fallback) is actually a way to limit growth of the renewable energy sector.
Increasing renewable penetration into a power system leads to:
-> reduced inertia
-> increasing Rate of Change of Frequency (RoCoF)
-> reduced time to react
If you rely entirely on manual systems to react to frequency changes on a grid with high renewable energy penetration (e.g. the Irish grid), you'd really struggle to avoid daily blackouts without decommissioning a bunch of wind farms.
It is by using such automated control schemes that countries like Ireland are able to push the limit on wind power penetration.
If I was being truly cynical, I would think that a push towards mandating manual control (rather than mandating that it is available as a fallback) is actually a way to limit growth of the renewable energy sector.
While a smart grid is incredibly useful to solve this, is it strictly necessary?
For example you could build an array of flywheels that charges when the frequency is above 50Hz and discharges when the frequency is below 50Hz.
No network connection necessary, and if you want to you can build it entirely analog. If all you really want is inertia and reaction time it could even be as simple as dumb three phase motors with weights.
For example you could build an array of flywheels that charges when the frequency is above 50Hz and discharges when the frequency is below 50Hz.
No network connection necessary, and if you want to you can build it entirely analog. If all you really want is inertia and reaction time it could even be as simple as dumb three phase motors with weights.
> For example you could build an array of flywheels that charges when the frequency is above 50Hz and discharges when the frequency is below 50Hz.
All rotary field machines (power plant generators, phase shifters, motors) work this way and contribute stability to the grid.
These basically add inertia (i.e. dampen frequency changes), but they do not regulate frequency. That's a fundamental difference.
All rotary field machines (power plant generators, phase shifters, motors) work this way and contribute stability to the grid.
These basically add inertia (i.e. dampen frequency changes), but they do not regulate frequency. That's a fundamental difference.
A simple motor or generator simply add inertia, but I see no reason why you can't use a flywheel to deliberatly regulate frequency, for example by adding a continuously variable transmission.
But really inertia is all you need because all we are trying to do here is replacing the lost inertia from replacing spinning generators with solar. The recovered inertia gives human operators the time to make phone calls to increase energy production, spinning up a pumped-hydro plant or whatever is available.
But really inertia is all you need because all we are trying to do here is replacing the lost inertia from replacing spinning generators with solar. The recovered inertia gives human operators the time to make phone calls to increase energy production, spinning up a pumped-hydro plant or whatever is available.
Who is going to pay to build and operate this flywheel inertia plant that generates no power? Sure it is possible but if you are going to spend that kind of money you can do a lot better.
It's been done already! Beacon Power operates a 20 MW / 5 MWh flywheel plant in Stephenstown, NY that does nothing but store grid energy and release it when needed. And ancillary services such as spinning reserve carry a price on the electricity market, so there is definitely an opportunity there.
I'm not sure if frequency regulation is currently considered an ancillary service in electricity markets right now though - from what I know about NY state, only 10- and 30-minute reserve are priced.
I'm not sure if frequency regulation is currently considered an ancillary service in electricity markets right now though - from what I know about NY state, only 10- and 30-minute reserve are priced.
The same people that would alternatively pay for smart grids: grid operators tha want to prevent blackouts in a changing environment. It's just another piece of infrastructre next to power lines and countless substations and transformers.
Usually power plant generators are required to respond to system frequency changes with a 5% droop characteristic so that they do regulate frequency by arresting dips and spikes.
We can do analog control. Almost anything digital can do, analog can as well... although often with very complicated implementations.
I'd consult the relevant project drawdown entry https://www.drawdown.org/solutions/electricity-generation/gr...
[deleted]
This.
As someone responsible for the IT systems reliability of a grid operator I have to wonder if this really is a backhanded attempt to limit to the growth of renewables.
We run load frequency control on the grid every 4 seconds to keep the grid in balance. Hard to see how that'd work with manual controls on today's complex grid.
As someone responsible for the IT systems reliability of a grid operator I have to wonder if this really is a backhanded attempt to limit to the growth of renewables.
We run load frequency control on the grid every 4 seconds to keep the grid in balance. Hard to see how that'd work with manual controls on today's complex grid.
Or, nuclear.
If we didn't let scaremongerers ruin nuclear with politics we could have had a 100% carbon-neutral (I don't count transportation because we can move to electric vehicles) grid with more cheap power available than we could find a use for by now.
Nuclear is statistically the safest form of energy in terms of deaths/kWh and waste storage is massively overblown, not even accounting for how we could be recycling that waste in new plant designs if it weren't for the fubar politics around it.
Nuclear is statistically the safest form of energy in terms of deaths/kWh and waste storage is massively overblown, not even accounting for how we could be recycling that waste in new plant designs if it weren't for the fubar politics around it.
> Nuclear is statistically the safest form of energy in terms of deaths/kWh
This is somewhat misleading, because the major risks are things like birth defects and cancers which are really hard to attribute to any specific cause. Not to mention that it's stochastic; we probably won't have a disaster, but if we do it ruins a huge land area at tremendous cost.
Also, this doesn't work for every country.
This is somewhat misleading, because the major risks are things like birth defects and cancers which are really hard to attribute to any specific cause. Not to mention that it's stochastic; we probably won't have a disaster, but if we do it ruins a huge land area at tremendous cost.
Also, this doesn't work for every country.
Exactly, according to Russia, only 31 people died as a result of Chernobyl.
does modern nuclear eschew digital controls? i mean... i’m sure they have manual cutoffs and sensors that humans monitor; can’t have some rogue hacker able to turn a nuclear plant into a huge catastrophe from the other side of the world and all...
but i’d think they’d be largely automated to reduce human error?
but i’d think they’d be largely automated to reduce human error?
Parent was talking about the necessity of networked digital control systems to manage intermittent renewable sources.
My point was that a grid comprised of a higher percentage of nuclear wouldn't need as advanced networked feedback, as sources wouldn't be turning on and off suddenly.
Indeed there would be local digital monitoring and control with nuclear, but that's a much less risky threat vector.
My point was that a grid comprised of a higher percentage of nuclear wouldn't need as advanced networked feedback, as sources wouldn't be turning on and off suddenly.
Indeed there would be local digital monitoring and control with nuclear, but that's a much less risky threat vector.
Digital elements in control is not the same as internet controlled.
There is no valid reason to make nuclear power plant controls accessible for control by internet.
There is no valid reason to make nuclear power plant controls accessible for control by internet.
It would be crazy to make it network-controllable, but you could have a data diode to allow power generation metrics and other monitoring data pass from the control network to the outside world.
I agree that the grid needs to handle variable sources and storage units, but how much of the problem can be handled by storage units? Molten salt storage looks better every day and it can be deployed anywhere with simple equipment.
https://www.energy.gov/articles/department-energy-announces-...
Physical grid improvements may be as important as computational ones. More available and cheaper electricity is crucial to reducing the use of gas-powered building heat systems, which represent a large chunk of CO2 emissions:
https://www.eesi.org/files/climate.pdf
https://www.energy.gov/articles/department-energy-announces-...
Physical grid improvements may be as important as computational ones. More available and cheaper electricity is crucial to reducing the use of gas-powered building heat systems, which represent a large chunk of CO2 emissions:
https://www.eesi.org/files/climate.pdf
US Gov taking steps to prevent the Butlerian Jihad before it even begins.
Good thinking security wise but could be a serious setback for efficiency
They could still collect data locally and use ML to optimize it.
Efficiency does not trump security for such installations.
It may be an unpopular opinion but I disagree - security is a cognitively biased function owing to historical roots but it actually seldom comes up which fails to justify its costs and there is little incentive to gain from attacking it.
Even if a bad actor did bring down the grid all it would do to concrete capabilities is GDP damage that they could easily get credit for and still leave a pissed off superpower able to make an example. Even actual rival status like say China were to do it would be shooting themselves in the foot such that something stupid and self destructive that if the PRC decided to simply start bombing Beijing it would do less damage.
Besides for decades a few tens or hundreds of people could go out, buy semi automatic rifles and start casually shooting substations and walking or driving away until someone arrests them. That would cripple power infastructure far worse than any hacker could as it would take longer to replace and be a distribuited issue to fix.
Given actual frequency even with lax security and pessimistic assumptions efficiency is a likely winner. Logistics and not superweapons win wars.
Even if a bad actor did bring down the grid all it would do to concrete capabilities is GDP damage that they could easily get credit for and still leave a pissed off superpower able to make an example. Even actual rival status like say China were to do it would be shooting themselves in the foot such that something stupid and self destructive that if the PRC decided to simply start bombing Beijing it would do less damage.
Besides for decades a few tens or hundreds of people could go out, buy semi automatic rifles and start casually shooting substations and walking or driving away until someone arrests them. That would cripple power infastructure far worse than any hacker could as it would take longer to replace and be a distribuited issue to fix.
Given actual frequency even with lax security and pessimistic assumptions efficiency is a likely winner. Logistics and not superweapons win wars.
Instead of fixing the laws and forcing proper auditing there should rather be regression in scalability, response and monitoring?
Very good news.
Mandate paper ballots next please.
Mandate paper ballots next please.
Punch cards. Scantrons. Stone Age shit for elections.
Anything that the public depends on for survival could be a great candidate for this type of restriction. Think of the hundreds of lives that would have been saved had this type of law been in place for airlines and their aircraft. A guy aware of the risks running himself into a tree in a Tesla is one thing, an unaware set of pilots with hundreds of passengers on a public aircraft is another.
Some accidents caused by automation failures would have been avoided, but how many others would have been caused by human failures?
If the automation is actively saving lives by correcting human failures then they need to log those events silently and forward them to whoever manages the pilots.
Since 1970, air travel has increased by a factor of 7, while fatalities have decreased by >80%. Combine these figures and you get a safety improvement by a factor of about 70.
We are clearly doing something right, and automated systems replacing error-prone humans may well be a part of it. I don’t quite get HN’s infatuation with the mythical perfect pilot.
We are clearly doing something right, and automated systems replacing error-prone humans may well be a part of it. I don’t quite get HN’s infatuation with the mythical perfect pilot.
This is very good news. Maybe few years late but better now. Current situation is a disaster waiting to happen
Stuxney penetrated a heavily guarded, monitored, secured with air gap facility. US power grid systems are child’s play in comparison
Stuxney penetrated a heavily guarded, monitored, secured with air gap facility. US power grid systems are child’s play in comparison
This is a huge indictment of the IT industry. We need to kick the shiny rock mentality, and actually engineer systems that are safe and secure.
Just look at how some companies interview and select people. They do so on the basis of cleverness, not carefulness and attention to detail.
Just look at how some companies interview and select people. They do so on the basis of cleverness, not carefulness and attention to detail.
Even if software were perfect, this makes sense as a defense-oriented decision. It's harder to hack a physical switch. Air gaps can be bridged.
There are also plenty of reasons to automate things. It allows for for faster reactions to problems and interesting new ways to reroute power. Both approaches have pros and cons, and there are good reasons for backing either approach.
Russia attacked Ukraine in a new way, and we responded by trying to become more well-defended against that new attack. Much as I agree that coding interviews are problematic, I fail to see how your point follows.
There are also plenty of reasons to automate things. It allows for for faster reactions to problems and interesting new ways to reroute power. Both approaches have pros and cons, and there are good reasons for backing either approach.
Russia attacked Ukraine in a new way, and we responded by trying to become more well-defended against that new attack. Much as I agree that coding interviews are problematic, I fail to see how your point follows.
The values that we hold in the industry are often the very things that lead to problems. We love generic things that can flexibly serve multiple purposes. And then we end up with a data port (USB) that is also used as a power source. What could possibly go wrong if I plug my computer to get power, but that same connection can be used to attack my computer? We love the cloud, but what could possibly go wrong if I hook my front door lock up to it? If the IT industry built cars, rolling down the window would occasionally cause the undocumented ejection seat to kill the passenger.
It is possible to write safe and secure software. It's expensive, but it's possible.
Digitizing the grid has enormous upside, to the tune of billions in savings and improved resiliency/response to weather related outages. It we could do it safely and securely it'd be a no-brainer.
We're just trading a devil we know for a (preventable) devil we don't.
BTW: digitized grids aren't even necessarily more vulnerable. In a complex system, the increased latency and miscommunication opportunities introduced by human operators are also a potential attack vector...
Digitizing the grid has enormous upside, to the tune of billions in savings and improved resiliency/response to weather related outages. It we could do it safely and securely it'd be a no-brainer.
We're just trading a devil we know for a (preventable) devil we don't.
BTW: digitized grids aren't even necessarily more vulnerable. In a complex system, the increased latency and miscommunication opportunities introduced by human operators are also a potential attack vector...
Digitizing the grid also opens up efficiency gains which we sorely need. Smart water heaters which run when we have daytime solar surplus, etc.
[deleted]
Why not send a price signal separate from the grid, or use AC frequency deviation as a price signal?
The neat thing is that already basically exists in the form of voltage.
If you've ever watched your voltage, you'd noticed that it isn't a perfect 110 or 220. It is often higher or lower. When it is higher, there is a local surplus, when it is lower, there is a high load.
We could do this today. We might not have current pricing, but we do have load vs production information.
If you've ever watched your voltage, you'd noticed that it isn't a perfect 110 or 220. It is often higher or lower. When it is higher, there is a local surplus, when it is lower, there is a high load.
We could do this today. We might not have current pricing, but we do have load vs production information.
> When it is higher, there is a local surplus, when it is lower, there is a high load.
Or perhaps the voltage got too low, and an on-load tap changer in one of the transformers increased the output voltage. Voltage does not necessarily follow the load. AFAIK, the thing generators themselves use as the main feedback signal is not voltage, but frequency; but it's not a useful signal for consumers, given that generators are much stronger at keeping the frequency at its nominal value.
Or perhaps the voltage got too low, and an on-load tap changer in one of the transformers increased the output voltage. Voltage does not necessarily follow the load. AFAIK, the thing generators themselves use as the main feedback signal is not voltage, but frequency; but it's not a useful signal for consumers, given that generators are much stronger at keeping the frequency at its nominal value.
Frequency doesn't change with load.
Load causes the voltage to drop (that's what's happening when a "brown out" is triggered). Some loads cause the current to lead or lag the voltage wave (Inductive vs Capacitive loads, most are Inductive, particularly with heavy duty equipment). But that isn't changing the frequency but rather the phase of the current. This is all tied up with a number referred to the "Power factor" (see https://en.wikipedia.org/wiki/Power_factor ). essentially, the farther shifted current is from voltage, the more work is done by the power plants essentially heating grid wires (rather than doing something useful)
So, power grids will do 2 things. First, they'll work to keep the current and voltage phase in sync. They do this by adding extra capacitors/inductors.
Second, they work to maintain the voltage of their tie in to to the grid.
Generally speaking, the type of power plant matters as well. Base load plants will simply dump onto the grid at a constant rate (without really caring about what the voltage is) while peaker and load following plants will attempt to vary output relative to their voltage to try and keep the grid voltages stable.
You are correct, the voltage variance can be misleading at the customer level if the transformer is actively adjusting it's voltage ratio. I didn't consider that.
Load causes the voltage to drop (that's what's happening when a "brown out" is triggered). Some loads cause the current to lead or lag the voltage wave (Inductive vs Capacitive loads, most are Inductive, particularly with heavy duty equipment). But that isn't changing the frequency but rather the phase of the current. This is all tied up with a number referred to the "Power factor" (see https://en.wikipedia.org/wiki/Power_factor ). essentially, the farther shifted current is from voltage, the more work is done by the power plants essentially heating grid wires (rather than doing something useful)
So, power grids will do 2 things. First, they'll work to keep the current and voltage phase in sync. They do this by adding extra capacitors/inductors.
Second, they work to maintain the voltage of their tie in to to the grid.
Generally speaking, the type of power plant matters as well. Base load plants will simply dump onto the grid at a constant rate (without really caring about what the voltage is) while peaker and load following plants will attempt to vary output relative to their voltage to try and keep the grid voltages stable.
You are correct, the voltage variance can be misleading at the customer level if the transformer is actively adjusting it's voltage ratio. I didn't consider that.
This is exactly how it's already done :)
https://en.m.wikipedia.org/wiki/Zellweger_off-peak
https://en.m.wikipedia.org/wiki/Zellweger_off-peak
> It is possible to write safe and secure software. It's expensive, but it's possible.
And we've done it before (safe, anyway). Is our electric grid as critical as space shuttle software?
And we've done it before (safe, anyway). Is our electric grid as critical as space shuttle software?
Maybe not the space shuttle, but:
NASA’s Mars Climate Orbiter; crashed or is now inoperable and orbiting the sun due to a bad numerical conversion.
ESA’s Ariane 5 Flight 501; manual self destruct triggered after a 64bit number being truncated into 16bits caused faults to be thrown.
https://raygun.com/blog/costly-software-errors-history/#
I don’t think we should go back to manual operation (as the default, but should be overridable). Instead we should be using stricter compilers, and better unit, integration tests, and fuzz testing to test as many edge conditions as possible.
NASA’s Mars Climate Orbiter; crashed or is now inoperable and orbiting the sun due to a bad numerical conversion.
ESA’s Ariane 5 Flight 501; manual self destruct triggered after a 64bit number being truncated into 16bits caused faults to be thrown.
https://raygun.com/blog/costly-software-errors-history/#
I don’t think we should go back to manual operation (as the default, but should be overridable). Instead we should be using stricter compilers, and better unit, integration tests, and fuzz testing to test as many edge conditions as possible.
Matt Parker's book Humble Pi is full of math/programming errors like this.
Not to be a pessimist, but just because a software error never caused a catastrophic space shuttle accident doesn't necessarily mean that the software actually was safe and secure.
> Just look at how some companies interview and select people. They do so on the basis of cleverness, not carefulness and attention to detail.
That's a direct reflection of how we develop software. New and shiny trumps reliable and boring any day of the week. That, coupled with an industry that does not even recognize the concepts of liability and defective product is a surefire recipe for a very large disaster at some point in the future. And after that it will get a lot easier to get budget for security, testing and all those other things that companies see as unnecessary cost.
That's a direct reflection of how we develop software. New and shiny trumps reliable and boring any day of the week. That, coupled with an industry that does not even recognize the concepts of liability and defective product is a surefire recipe for a very large disaster at some point in the future. And after that it will get a lot easier to get budget for security, testing and all those other things that companies see as unnecessary cost.
On the other hand, maybe the problem with our industry is that we try to solve every problem with computers and software when simpler and more secure physical solutions can do a better job in some cases.
Even old-school industries aren't immune. I had an embedded hiring manager ask me the burning stick, 45 minute timing question along with how many interrupts there were on a processor i used five years ago. A lot of engineers are doltish in general when assessing others.
I will probably quote this comment.
It does take both kinds of kinds... creative thinking for hard problems, but attention to detail towards the path of correctness. I completely agree the "cleverness" is massively over-emphasized during the interviews for most companies
It does take both kinds of kinds... creative thinking for hard problems, but attention to detail towards the path of correctness. I completely agree the "cleverness" is massively over-emphasized during the interviews for most companies
[deleted]
The attitude in the industry towards developers is pretty much:
You know Java? Cool. My kid is learning that in high school.
You know Java? Cool. My kid is learning that in high school.
You set the bar high. Some are more like, "Talk to this guy, he's a computer geek."
As a person with two electrical engineering degrees and 15 years' experience as a professional programmer, I am nearly speechless at such a rational, foresighted, and prudent action coming from our government, in regards technology. It's like some bizarro-world where we have leaders who use the power of government to encourage industry to be careful and secure. I will wake up soon, and realize this was all a dream.
I suppose a captured regulatory body is right twice a day
This put a genuine smile on my face =p
For some reason this reminded me about the still-unsolved San Jose substation physical attack. The shooters did a lot of damage but didn’t interrupt much power.
https://en.wikipedia.org/wiki/Metcalf_sniper_attack
https://en.wikipedia.org/wiki/Metcalf_sniper_attack
The opposite of "digital" is "analog", not "manual".
The old phone networks were analog and vulnerable to cereal box whistles.
The old phone networks were analog and vulnerable to cereal box whistles.
Analog can still be automated. The right word here is indeed manual since it needs human intervention.
Electronics also need to be protected against EM weapons.
Just hope this doesn't snowball
This is quite an indictment of the security community.
This is quite an indictment of the security community.
I have mixed feelings about this. In some ways it's a good idea but the demonization of "digital" is misguided. Some of us know how to build secure digital hardware and software but there are no incentives for companies to insist upon such designs, and there are many active disincentives for doing so. The incentive structure needs to be changed to fix this, and then the technology will follow. Demonizing a technology is short-sighted and dangerous.
This seems like a terrible idea. Computer controlled systems lead to optimization and smarter decisions.
Not always in my experience. It’s balanced with “Out of sight out of mind”. There’s a benefit to making certain critical decisions manual, as it forces some review and awareness.
Plus the current track records of internet security for physical control systems is terrible.
Plus the current track records of internet security for physical control systems is terrible.
should have used erlang?
"bill that will study ways to replace automated systems with low-tech redundancies to protect the country's electric grid from hackers."
The headline seems to be inferring more than the content is saying. The intent is not to rollback digitized systems, it's to decrease dependency on vulnerable systems. And the bill is only to commission a study on what systems could benefit from manual fallback, nothing is slated to be implemented yet.
The headline seems to be inferring more than the content is saying. The intent is not to rollback digitized systems, it's to decrease dependency on vulnerable systems. And the bill is only to commission a study on what systems could benefit from manual fallback, nothing is slated to be implemented yet.
Thank you for the only substantive comment here!
Alternatively, this could also be a misguided effort that holds back smart-grid tech in America by a decade. It's really hard to tell.