Final Report on the August 14, 2003 Blackout (2004) [pdf](energy.gov)
energy.gov
Final Report on the August 14, 2003 Blackout (2004) [pdf]
https://www.energy.gov/sites/prod/files/oeprod/DocumentsandMedia/BlackoutFinal-Web.pdf
17 comments
When the Tubbs Fire happened north of the SF Bay last year, it cut off the few Internet links to the Eureka area in Northern California. ATM machines were down as a result, as were the phone lines. If you were stuck somewhere without gas, and without cash, you were really out of luck.
This kind of thing can happen at any moment and yet how many people are really prepared?
This kind of thing can happen at any moment and yet how many people are really prepared?
Fact: 70% of residents in the 3 counties (Napa, Lake, Sonoma) had FULL telecom outage (no internet, phone, cell) for 2-3 days. Power was out for 2 days at our house.
When your entire profession and skillset is based on the internet, and computers, etc. -- and those things simply cease to exist for all practical purposes - that is a bad feeling.
When your entire profession and skillset is based on the internet, and computers, etc. -- and those things simply cease to exist for all practical purposes - that is a bad feeling.
The movie "Live Free or Die Hard" explores this idea in the context of terrorism - the plot revolves around a "fire sale" in which transportation, transcommunication, and power/utilities are effectively shut down. It is a Die Hard film though so it's not quite based in reality.
If they're petrol stations, why not just hook up a generator and charge 5% more ? Or, you know, 50%.
(sorry about the start of the video, but it shows the type of equipment you'd use for this sort of situation) https://www.youtube.com/watch?v=FiuNVTLHqEc
(sorry about the start of the video, but it shows the type of equipment you'd use for this sort of situation) https://www.youtube.com/watch?v=FiuNVTLHqEc
Presumably they (a) don't have a genset of sufficient power (b) don't have the required hookups to disconnect from the mains and plug the generator in and maybe (c) aren't keen on running a generator next to the "switch off engine" signs.
(The nearest petrol station to me is automated, so there would be nobody to do this anyway!)
(The nearest petrol station to me is automated, so there would be nobody to do this anyway!)
One of my favourite anecdotes about the 2003 blackout is this exchange[1]:
Other Operator: “Hey, do you think you could help out the 345 voltage a little?”
Eastlake 5 Operator: “Buddy, I am — yeah, I’ll push it to my max max. You’re only going to get a little bit.”
Other Operator: “That’s okay, that’s all I can ask.”
Eastlake Unit 5 tripped on overload shortly after this, removing reactive power capability from the system and further destabilizing it.
Human error and lack of situational awareness (Midwest ISO's state estimator, a software program that is supposed to continually evaluate power system conditions and alert operators when they are in or near a region of instability, was offline) played a big role.
[1] https://en.wikibooks.org/wiki/Professionalism/Northeast_Blac...
Other Operator: “Hey, do you think you could help out the 345 voltage a little?”
Eastlake 5 Operator: “Buddy, I am — yeah, I’ll push it to my max max. You’re only going to get a little bit.”
Other Operator: “That’s okay, that’s all I can ask.”
Eastlake Unit 5 tripped on overload shortly after this, removing reactive power capability from the system and further destabilizing it.
Human error and lack of situational awareness (Midwest ISO's state estimator, a software program that is supposed to continually evaluate power system conditions and alert operators when they are in or near a region of instability, was offline) played a big role.
[1] https://en.wikibooks.org/wiki/Professionalism/Northeast_Blac...
I remember this blackout. I was in college at the time and this was a big deal back in Long Island. A lot of us got together and had a lot of fun in the dark as everything was shut down including labs, libraries. It turned out to be a great fun time with friends while we sat and discussed our lives without power.
This is probably a dumb idea, but wouldn't it be cool to have scheduled blackouts like this from time to time?
A couple of us got flashlights and went to the main street and it was pitch dark and quiet and eerie. That was fun. Other than that, living without power got real boring, real quick.
TLDR; Caused by trees. And poor planning.
FirstEnergy didn't adequately trim trees in the right of way for a 345 kV line. On a hot day, due to other outages, the load on that line caused the conductor to heat, expand, sag, and then electricity shorted to the tree. Protection equipment removed the line from service, but with the loss of that transmission capacity, other lines also exceeded their capacity with their protection equipment tripping them offline. Planning criteria should have 1) prevented the first line from having as much load as it did, and 2) prevented the loss of the first line from causing cascading failures.
Page 18 is a good place to start if you don't have the patience for the full 238 page report.
FirstEnergy didn't adequately trim trees in the right of way for a 345 kV line. On a hot day, due to other outages, the load on that line caused the conductor to heat, expand, sag, and then electricity shorted to the tree. Protection equipment removed the line from service, but with the loss of that transmission capacity, other lines also exceeded their capacity with their protection equipment tripping them offline. Planning criteria should have 1) prevented the first line from having as much load as it did, and 2) prevented the loss of the first line from causing cascading failures.
Page 18 is a good place to start if you don't have the patience for the full 238 page report.
That's not a fully accurate situation.
The most damning thing is that FirstEnergy didn't realize their system was falling apart. They lost alarm notifications at 2:14, and didn't realize that things were bad for another hour and a half. The first transmission line to fail completely [1] wasn't discovered by anyone until after the blackout, and FirstEnergy was called about the first transmission line failure but they dismissed it as a fluke because they received no notification of its failure. It took several people calling them asking about line failures before they realized that they weren't seeing their internal notifications, and by that time, their system was pretty much one minor thing away from disaster.
The only actual recourse they would have had at that point would have been to start disconnecting people from their system. That takes time, and by the time they realized that they were in such an emergency, it was too late. The system was already collapsing and the full blackout was just 20 minutes away.
[1] The Star-South Canton transmission line crossed company boundaries and failed twice then immediately automatically reconnected before failing for good a third time. The first failure precedes the Harding-Chamberlain failure, while the other two failures occurred after two lines were off for good, so this line's failure is considered the third failure.
The most damning thing is that FirstEnergy didn't realize their system was falling apart. They lost alarm notifications at 2:14, and didn't realize that things were bad for another hour and a half. The first transmission line to fail completely [1] wasn't discovered by anyone until after the blackout, and FirstEnergy was called about the first transmission line failure but they dismissed it as a fluke because they received no notification of its failure. It took several people calling them asking about line failures before they realized that they weren't seeing their internal notifications, and by that time, their system was pretty much one minor thing away from disaster.
The only actual recourse they would have had at that point would have been to start disconnecting people from their system. That takes time, and by the time they realized that they were in such an emergency, it was too late. The system was already collapsing and the full blackout was just 20 minutes away.
[1] The Star-South Canton transmission line crossed company boundaries and failed twice then immediately automatically reconnected before failing for good a third time. The first failure precedes the Harding-Chamberlain failure, while the other two failures occurred after two lines were off for good, so this line's failure is considered the third failure.
Failure to trim wild growth is a very common cause of massive wildfires.[0]
In California, there's an incentive problem because the utilities try to charge the costs to the consumers. They have reduced economic incentive to maintain safe zones around their equipment.
[0] http://www.latimes.com/business/la-fi-utility-wildfires-2017...
In California, there's an incentive problem because the utilities try to charge the costs to the consumers. They have reduced economic incentive to maintain safe zones around their equipment.
[0] http://www.latimes.com/business/la-fi-utility-wildfires-2017...
FirstEnergy -- the same company currently gunning for a multi-billion bailout in the name of system reliability. :)
I'm still working through the report, but your TLDR seems wrong.
The report seems to be repeatedly emphasizing that the grid is supposed to be resilient to unpredictable failures at unpredictable times. While the trees were the immediate cause [0], the real story of the outage was the resulting cascade failure, which absolutly should not have happened.
To take a CS example, when Google/Amazon/etc suffer a cascade failure, they do not blame the proximal cause (such as a few nodes getting overloaded), but rather their load balances and failure recovery systems for allowing/causing the problem to cascade into a full network failure.
[0] And a result of inproper vegetation management
The report seems to be repeatedly emphasizing that the grid is supposed to be resilient to unpredictable failures at unpredictable times. While the trees were the immediate cause [0], the real story of the outage was the resulting cascade failure, which absolutly should not have happened.
To take a CS example, when Google/Amazon/etc suffer a cascade failure, they do not blame the proximal cause (such as a few nodes getting overloaded), but rather their load balances and failure recovery systems for allowing/causing the problem to cascade into a full network failure.
[0] And a result of inproper vegetation management
i was in the basement of a largish building. our office was down there, and we'd got used to the hum of all the internals at the other end of the building (hvac mostly, but...other ambient power noises in general).
4ish pm (ann arbor area)... boom. no power. pitch black. in a basement. with no sounds. no ambient hum of the hvac. nothing, save a single red 'exit' emergency light. we made our way upstairs and outside, and ... yep - the whole town was shut down. took a while to realize it was a good portion of the northeast!
looking back, I was quite surprised no one was caught in an elevator. there were only 4 floors, and it was 4pm, so things were slow, but... they did get used regularly.
4ish pm (ann arbor area)... boom. no power. pitch black. in a basement. with no sounds. no ambient hum of the hvac. nothing, save a single red 'exit' emergency light. we made our way upstairs and outside, and ... yep - the whole town was shut down. took a while to realize it was a good portion of the northeast!
looking back, I was quite surprised no one was caught in an elevator. there were only 4 floors, and it was 4pm, so things were slow, but... they did get used regularly.
My mother told me later how suddenly terrifying the situation became.
There is no power. Your car may not have enough fuel to get home to your kids. We do not know when there will be power again. It may be in an hour, or next week. You are 100 miles from home- a distance that used to be an inconvenience but suddenly might be an impossibility. The phone system may not be working either, or may not for very much longer. You have whatever money you're holding in cash right now to pay for your needs.
We are dependent on our technology in ways we cannot even see.
(They pulled into the driving running on the last fumes in the tank, fortunately)