The Colonial Pipeline Was Fine, Shutdown to Make Sure Its Owner Would Get Paid(jalopnik.com)
jalopnik.com
The Colonial Pipeline Was Fine, Shutdown to Make Sure Its Owner Would Get Paid
https://jalopnik.com/the-colonial-pipeline-was-fine-but-it-was-shutdown-to-1846911689
105 comments
I think maybe you have to think about the total economic impact for the article and frustration to make more sense. Fuel is a very fundamental part of our economy and almost always goes toward producing much more value than the value of the fuel itself. Add to that, shortages can and do contribute to irreversible losses like loss of life. If every gas distributor shut down because ransomware hit them all, you could completely grind the country to a halt.
It's not too far fetched either to hope that their customers would be willing to self-report. Sure you might get some people who'd take advantage of the situation, but it's not a the total loss scenario of giving away your product for free that you're talking about. Add in the opportunity cost and shutting down delivery if you have alternatives that may not be 100% accurate billing is probably a net loss for the company, though it's impossible to say.
> I'm still trying to find out what actually happened in Israel but god forbid a news source or commentator actually lay out the details of a complicated situation... (if you're not getting everybody pissed off about X, then you're not doing your job as a reporter?)
It's a bit ironic to take a morals-over-profit stance on reporting right after having taken the profit-over-morals stance of gas pipelines.
It's not too far fetched either to hope that their customers would be willing to self-report. Sure you might get some people who'd take advantage of the situation, but it's not a the total loss scenario of giving away your product for free that you're talking about. Add in the opportunity cost and shutting down delivery if you have alternatives that may not be 100% accurate billing is probably a net loss for the company, though it's impossible to say.
> I'm still trying to find out what actually happened in Israel but god forbid a news source or commentator actually lay out the details of a complicated situation... (if you're not getting everybody pissed off about X, then you're not doing your job as a reporter?)
It's a bit ironic to take a morals-over-profit stance on reporting right after having taken the profit-over-morals stance of gas pipelines.
> It's not too far fetched either to hope that their customers would be willing to self-report. Sure you might get some people who'd take advantage of the situation
Would you bet millions of dollars on the goodness of the typical person?
Would you bet millions of dollars on the goodness of the typical person?
> Would you bet millions of dollars on the goodness of the typical person?
This is about business to business, not business to consumer. I suggest to talk to a financial/accounting person in a big company. Often enough there are mistakes in payments that are only noticed by the other business. This despite all the various layers that should've prevented the entire problem.
I don't mean small mistakes either, I mean where someone wanted to enter something like 1234.56 (two decimals), but it ended up as 1234560.00. Basically a mixup between the decimal and thousand separator; such mistakes happen way more often than you'd think. I've heard various stories of a vendor notifying the company that they've been overpaid, e.g. the 1000 times what it should've been, but also various other kinds of mistakes.
Some companies have been working together for various decades. Employees might've switched between the companies, people know each other. It really isn't uncommon to have quite a bit of trust between companies. Obviously, this does depend on the country. In some parts of the world there's more trust than other parts.
This is about business to business, not business to consumer. I suggest to talk to a financial/accounting person in a big company. Often enough there are mistakes in payments that are only noticed by the other business. This despite all the various layers that should've prevented the entire problem.
I don't mean small mistakes either, I mean where someone wanted to enter something like 1234.56 (two decimals), but it ended up as 1234560.00. Basically a mixup between the decimal and thousand separator; such mistakes happen way more often than you'd think. I've heard various stories of a vendor notifying the company that they've been overpaid, e.g. the 1000 times what it should've been, but also various other kinds of mistakes.
Some companies have been working together for various decades. Employees might've switched between the companies, people know each other. It really isn't uncommon to have quite a bit of trust between companies. Obviously, this does depend on the country. In some parts of the world there's more trust than other parts.
That's how credit card processing operated worldwide for many decades. That's also how hawala works. And Wikipedia, if you want a non-monetary example.
They are all literally trusting in the goodness of the typical person. And then doing some careful analysis and planning. And fraud detection for the rare misdeeds.
They are all literally trusting in the goodness of the typical person. And then doing some careful analysis and planning. And fraud detection for the rare misdeeds.
But a credit card still keeps track of how much you spend - it doesn't rely on you honestly reporting how much you spent over the month...
Oh, I think they are talking about the earlier period where you just did a carbon copy of the card, got it signed, and if the card was a fake there was certainly no way to tell at the time. It would be the equivalent of a bad check, talking the days before swipe machines.
I guess, but discounting active fraud, there is still a paper trail of how much you spent; The bank will still have said "We think you owe us $100, here is our reasoning based on things you signed" never "you spent some amount of money we're not sure of - please give it to us" or "all of our customers spent $1million, so let us have your share, whatever you think that may be" as would be the case here.
Yes, the system would fall apart if you disputed every single transaction and made them supply irrefutable evidence you did in fact authorise the transaction - but it doesn't revolve entirely on the trust that people will keep their own account of what they think they spent.
Yes, the system would fall apart if you disputed every single transaction and made them supply irrefutable evidence you did in fact authorise the transaction - but it doesn't revolve entirely on the trust that people will keep their own account of what they think they spent.
Sure, but the store that accepted the invalid credit card has traditionally been on the hook. The store is the one being targeted even if the person whose credit card was stolen can get off with disputing the charge.
And before everything was networked, there was no way for a store to check with the credit card company that the card wasn't reported stolen. That's the trusting people part, it's the store that is trusting people without the ability to even verify that the card hasn't been reported stolen.
Yes, there is a paper trail, but the situation I'm talking about is analogous to a bad check or a counterfeit bill -- there's no recourse if the bad actor can't be found and the store is on the hook.
And before everything was networked, there was no way for a store to check with the credit card company that the card wasn't reported stolen. That's the trusting people part, it's the store that is trusting people without the ability to even verify that the card hasn't been reported stolen.
Yes, there is a paper trail, but the situation I'm talking about is analogous to a bad check or a counterfeit bill -- there's no recourse if the bad actor can't be found and the store is on the hook.
Ah I understand now - from the shop’s point of view yes there is risk in this.
I am just about old enough to remember this time - it’s funny to think that security hinged on checking if the signature looked ‘right’. For cheques I remember here at least for yours to actually be accepted anywhere you needed a ‘cheque guarantee card’ which was basically just a debit card but with a fancy hologram - so a certificate of your ‘goodness’ - up to a certain limit per cheque. The shop would copy the details and this would supposedly guarantee they’ll be paid.
I was going to say that shops could have asked for ID - but at this time driving licences didn’t have a picture and weren’t cards - so no one carried them anyway. There was probably a lot of ‘subjective’ acceptance around the place (‘does this person look trustworthy, would they really be given a credit card or did they steal it?’) come to think of it…
I am just about old enough to remember this time - it’s funny to think that security hinged on checking if the signature looked ‘right’. For cheques I remember here at least for yours to actually be accepted anywhere you needed a ‘cheque guarantee card’ which was basically just a debit card but with a fancy hologram - so a certificate of your ‘goodness’ - up to a certain limit per cheque. The shop would copy the details and this would supposedly guarantee they’ll be paid.
I was going to say that shops could have asked for ID - but at this time driving licences didn’t have a picture and weren’t cards - so no one carried them anyway. There was probably a lot of ‘subjective’ acceptance around the place (‘does this person look trustworthy, would they really be given a credit card or did they steal it?’) come to think of it…
Yes, I didn't want it to veer off topic, but I think it is extremely obvious that such standards involved implicit or explicit biases, as well as real recognition of individual customers' actual past behavior (like if a gas station has taken advantage of them in the past).
I assume the companies then just passed the risk onto the consumer (or ate it to grow their user base), I find it hard to believe they just continued operating on the honor system.
Would you bet millions of dollars on payment system with zero redundancy? Also with that kind of money involved I wouldn't be surprised if you could could get a court case running and force them to report how much they took.
I mean that’s how lots of business works? Is that not the whole basis of invoicing? I must be misunderstanding
An invoice would say “you owe $x for y gallons of gas” not “hey pay us some money for that gas you got”
True, but still: a lot of (big) businesses are still quite unhappy if they didn't pay what they should've paid. It also creates an uncertainty for them, there's often various years where the money can be requested back. Plus as far as I know, if you notice an error there's various financial rules you need to follow. You cannot just treat it as "cost saved". As a result, it is way easier to notify the other party and get a mistake fixed.
I think there's a way they could count the number of all the trucks without a computer.
When all parties are losing a bunch of money on acount of the shutdown, the rational choice would be to accept some lesser risk than these known losses in order to get things working to some degree, if that were technically feasible.
It shouldn't even be up to them to make a rational choice. It should be required via federal regulation on utilities.
No, I would create legislation that mandates that the pipeline must keep running in situations like this at the company's expense
> Would you bet millions of dollars on the goodness of the typical person?
Somewhere in that question is an answer to the Fermi paradox!
Somewhere in that question is an answer to the Fermi paradox!
>It's not too far fetched either to hope that their customers would be willing to self-report.
Spoken like someone who has never done business.
Spoken like someone who has never done business.
Spoken like someone who would let critical public infrastructure get privatized by a corporate monopoly.
Somewhat of a moot point when said infrastructure is already private. You can use the misaligned incentives to argue against private ownership of infrastructure, but criticizing a business for doing business is barking up the wrong tree.
It's not a moot point if we can create regulation or government takeover of said infrastructure. Conversations like this are the first step towards creating those kinds of regulations. It's only moot if you make it moot. When a private business completely controls a public utility and fails to act in the public's interest it is time to have this conversation because they have their hands all over something that should be in the hands of the public.
My point is that any business would have done the same. Expecting one to not look out for its own bottom line and rely on the kind heartedness of its customers is misguided. The problem is with the game, not the player.
[deleted]
I wonder if there is a capitalistic approach so solving this problem (making incentives align between the needs of the fuel distributor and that of the economy in the case of technical failure)?
The only approach I see is perhaps to have written into contracts with all buyers some kind of compensation for the pipeline being unavailable per hour.
Then colonial will prefer to have the pipeline turned on handing out free fuel because it's cheaper than the compensation for turning off.
The only approach I see is perhaps to have written into contracts with all buyers some kind of compensation for the pipeline being unavailable per hour.
Then colonial will prefer to have the pipeline turned on handing out free fuel because it's cheaper than the compensation for turning off.
If only the world economy ran on tears and sympathy instead of recorded transactions. I should pray that all my utility and food providers function without charging me money... until they go bankrupt and I am left with neither.
You'd probably be pretty pissed if your electricity provider suffered a ransomware attack on its billing system and then decided to stop your heating completely on its own accord because it can't know if you're paying your fair share.
They do that anyways here even without ransomware attacks. There were all kinds of articles about it on the front page of HN.
When?
Texas recently shut down their power grid when it got cold out because it was cheaper than weatherproofing or suffering the regulations to interconnect with other neighbouring states.
That's not in any way similar. The Colonial Pipeline was not shut down because they ran out of petroleum to distribute.
Grandparent commenter with "Austin" in his name was saying his power gets shut off even w/out ransomware. Parent commenter asked "when?" and I answered the question.
I'm not taking a position on the similarity or dissimilarity between shutting down a service because you can't bill accurately and shutting down a service because you refuse to interoperate with other providers.
All I'm saying is Texas customers did get their electricity cut off recently.
I'm not taking a position on the similarity or dissimilarity between shutting down a service because you can't bill accurately and shutting down a service because you refuse to interoperate with other providers.
All I'm saying is Texas customers did get their electricity cut off recently.
The Texas power grid didn’t run out of fuel either. It was known there needed to be controlled outages a couple weeks in advance. What wasn’t known was the scale or duration of the problem.
At any rate you are just looking for something to cry about. You asked for a hypothetical counter example not expecting to get one and now are splitting hairs to qualify big tears.
At any rate you are just looking for something to cry about. You asked for a hypothetical counter example not expecting to get one and now are splitting hairs to qualify big tears.
>The Texas power grid didn’t run out of fuel either.
"As of Wednesday morning, when the power outages were at their most severe, the cold had snuffed out about 46 gigawatts, or about 40 percent, of power-generation capacity in the state."
Sounds a lot like the grid did in fact run out of "fuel" (as in the product that they deliver).
>You asked for a hypothetical counter example
Of an electricity provider cutting power of its own accord because their billing service was not functioning, not because the electricity production in Texas dropped due to freezing while demand shot up.
"As of Wednesday morning, when the power outages were at their most severe, the cold had snuffed out about 46 gigawatts, or about 40 percent, of power-generation capacity in the state."
Sounds a lot like the grid did in fact run out of "fuel" (as in the product that they deliver).
>You asked for a hypothetical counter example
Of an electricity provider cutting power of its own accord because their billing service was not functioning, not because the electricity production in Texas dropped due to freezing while demand shot up.
There are some restrictions on when the utilities can shut off your power, water, electricity, etc. Whether gasoline should be regulated like a utility is an interesting question. Given how dependent on cars most Americans are, seems like a reasonable conversation to have.
Frankly, you seem to lack knowledge here. Your example of a grocery store is B2C with no knowledge of the customers. The pipeline is B2B with legal contracts in place and use invoices.
You can order dozens of types of oil through the pipeline. With travel distances of thousands of miles, many destinations, and the need for product segregation, the pipeline knows where all the oil is going.
If they couldn’t track segregation and destinations in the pipeline, that is a good reason to shut down. If they were just going to be delayed in sending out invoices or had to input some items into the system manually after recovery, they most definitely should not have shut down.
You can order dozens of types of oil through the pipeline. With travel distances of thousands of miles, many destinations, and the need for product segregation, the pipeline knows where all the oil is going.
If they couldn’t track segregation and destinations in the pipeline, that is a good reason to shut down. If they were just going to be delayed in sending out invoices or had to input some items into the system manually after recovery, they most definitely should not have shut down.
> The pipeline is B2B with legal contracts in place and use invoices.
What if billing is tied to metered usage and it was impossible to meter? Or what if the billing system is needed to shut off access when customer has reached a quota? The article is scant on details but "legal contracts and invoices" are not going to save a company legal fees with when multiple customers dispute their bills.
What if billing is tied to metered usage and it was impossible to meter? Or what if the billing system is needed to shut off access when customer has reached a quota? The article is scant on details but "legal contracts and invoices" are not going to save a company legal fees with when multiple customers dispute their bills.
>If they couldn’t track segregation and destinations in the pipeline, that is a good reason to shut down.
Wouldnt that then conceded the pipeline was not fine?
>If they were just going to be delayed in sending out invoices or had to input some items into the system manually after recovery, they most definitely should not have shut down.
I am willing to bet a signifigant number of "customers" (businesses in this case with teams of lawyers) would challenge the vailidity of invoices based on estimates and not actual usage. Theres no free lunch, the pipeline owners understand this and decided its in the best interest for their company to stop deliveries until the system that manages their order tracking is back online. Thats totally reasonable.
The US government (and states for that matter) made 0 guarantees of coverage of any thing. And even if they did, its not like the government does much to make anything a straightforward process.
Wouldnt that then conceded the pipeline was not fine?
>If they were just going to be delayed in sending out invoices or had to input some items into the system manually after recovery, they most definitely should not have shut down.
I am willing to bet a signifigant number of "customers" (businesses in this case with teams of lawyers) would challenge the vailidity of invoices based on estimates and not actual usage. Theres no free lunch, the pipeline owners understand this and decided its in the best interest for their company to stop deliveries until the system that manages their order tracking is back online. Thats totally reasonable.
The US government (and states for that matter) made 0 guarantees of coverage of any thing. And even if they did, its not like the government does much to make anything a straightforward process.
It's understandable why they would shut it down temporarily if they could no longer accurately bill customers for usage.
At some point, however, it starts impacting the economy and there would have to measures in place to get the fuel going regardless of whether the corporate bean-counters (and their malware infested computers) were fully "ready". I don't know what that point in time should be. But the fact that there were news stories of imbeciles hoarding gasoline and emptying fully loaded gas stations within a couple of hours certainly makes it urgent to get the pipeline running sooner. Can't we all just get along?
At some point, however, it starts impacting the economy and there would have to measures in place to get the fuel going regardless of whether the corporate bean-counters (and their malware infested computers) were fully "ready". I don't know what that point in time should be. But the fact that there were news stories of imbeciles hoarding gasoline and emptying fully loaded gas stations within a couple of hours certainly makes it urgent to get the pipeline running sooner. Can't we all just get along?
>Yes, in fact, if a store's computers go down they don't just let customers walk out the door with products and hope they get paid eventually?
They better do that if they get the permission from the gov't to operate the only grocery store in the area, and people weren't able to get food otherwise.
They better do that if they get the permission from the gov't to operate the only grocery store in the area, and people weren't able to get food otherwise.
Okay, so establish some sort of "crisis service fund" that pays out when payment processing breaks through no fault of your own. This is something the state can do at any time. Honestly, allowing a monopoly without planning something like this just seems careless.
Providing service without hope of ever being paid is stupid. The market is pretty good at selecting against stupidity.
Providing service without hope of ever being paid is stupid. The market is pretty good at selecting against stupidity.
I lived both in a high-trust society and in a low-trust society. That you seem to have experienced only low-trust society may impact you worldview, but take my word for it, things are different elsewhere ;-)
Don't be too harsh. The experience of living in a high-trust society is becoming more and more rare for future generations, due to certain policy decisions. They will just assume things being broken is the natural order.
I really like this comment. It is an indictment of our society if we feel like we can't trust anyone around us. I have people around me with whom I'd trust my life; it is much better to surround ourselves with those people. And there are lots of examples of societies historically and now where things operated very largely on trust.
> Okay, so establish some sort of "crisis service fund" that pays out when payment processing breaks through no fault of your own.
One company which suffered a cyber attack solved this partly by transferring a rough estimate of what a vendor should've been paid as a bank transfer. This up front. They did similar things for smaller vendors (smaller vendors might have bigger issues not getting paid on time). Not for everyone due to the amount of work involved. They got agreements in place to sort out the exact amounts afterwards. This required a huge trust and at the same time it created a great amount of goodwill from those vendors. Meaning, these vendors took over work on behalf of the company.
Afterwards the exact costs can be figured out. What is important that business continues as much as possible, if you completely stop your business you'll also not have any revenue you could maybe charge later.
I think the pipeline action on first hand is a rather stupid action. Loads of business is pretty predictable, plus often you already had the insight on the orders for the coming period. The cost of some mishaps and the economic damage to the country should be considered as well. It seems a case of "penny wise, pound foolish".
Regarding "payment processing" breakdowns: there are various things in place in NL for that, though most don't know. Simply stated: you fill out a paper form, sign it, done.
One company which suffered a cyber attack solved this partly by transferring a rough estimate of what a vendor should've been paid as a bank transfer. This up front. They did similar things for smaller vendors (smaller vendors might have bigger issues not getting paid on time). Not for everyone due to the amount of work involved. They got agreements in place to sort out the exact amounts afterwards. This required a huge trust and at the same time it created a great amount of goodwill from those vendors. Meaning, these vendors took over work on behalf of the company.
Afterwards the exact costs can be figured out. What is important that business continues as much as possible, if you completely stop your business you'll also not have any revenue you could maybe charge later.
I think the pipeline action on first hand is a rather stupid action. Loads of business is pretty predictable, plus often you already had the insight on the orders for the coming period. The cost of some mishaps and the economic damage to the country should be considered as well. It seems a case of "penny wise, pound foolish".
Regarding "payment processing" breakdowns: there are various things in place in NL for that, though most don't know. Simply stated: you fill out a paper form, sign it, done.
Sounds like solving the problem from the wrong end. The payment infrastructure is the service provider's responsibility, not the state's.
Instead, set up strict SLAs with harsh fines; make it more expensive to shut down than to provide free service.
Instead, set up strict SLAs with harsh fines; make it more expensive to shut down than to provide free service.
There are ways of collecting payments that don’t involve contactless card terminals.
The other thing is when you have hackers in your network, shutting down the things that could be physically dangerous if they access it, until you know what you are facing and you identified the compromised machines, doesn't seem unreasonable to me.
That depends on what the alternative is. If it involves clueless people filling up garbage bags with gasoline and storing them in their kitchens perhaps letting the pipeline continue to run would have been safer.
it's also sounding kind of like the simplification an IT clueless manager might make after too much coffee and not enough sleep.
it whats likely happened was that the hardened industrial control network was pretty much fine meaning that engineers inside of the pumping facilities could manually direct/redirect flow with all the required safeguards still in place, but that they had no plan for where to direct the flow as all information about who needed/ordered what sat in the unhardened corporate network.
This is kind of similar to what happens to warehouses and "just in time" factory lines when the order printer/queue ceases up, or is no longer fed data from the back end system, and while one might expect that the operators of critical infrastructure have a plan for minimal services when/if the unhardened corporate network fails but that plan(even if it existed) likely sat on the same unsecured and now encrypted network.
it whats likely happened was that the hardened industrial control network was pretty much fine meaning that engineers inside of the pumping facilities could manually direct/redirect flow with all the required safeguards still in place, but that they had no plan for where to direct the flow as all information about who needed/ordered what sat in the unhardened corporate network.
This is kind of similar to what happens to warehouses and "just in time" factory lines when the order printer/queue ceases up, or is no longer fed data from the back end system, and while one might expect that the operators of critical infrastructure have a plan for minimal services when/if the unhardened corporate network fails but that plan(even if it existed) likely sat on the same unsecured and now encrypted network.
> Yes, in fact, if a store's computers go down they don't just let customers walk out the door with products and hope they get paid eventually?
It's one thing with a supermarket and anonymous masses of customers, but an oil pipeline where each customer is supposed to have meters running on their side as well (as double verification to protect against technical issues)? If you can't trust your known (!) customers enough to not rip you off, something is very much off.
And even in that case: the government could have told the pipeline owners "we'll pay off any differences after settling, now go and get that pipeline up and running again".
It's one thing with a supermarket and anonymous masses of customers, but an oil pipeline where each customer is supposed to have meters running on their side as well (as double verification to protect against technical issues)? If you can't trust your known (!) customers enough to not rip you off, something is very much off.
And even in that case: the government could have told the pipeline owners "we'll pay off any differences after settling, now go and get that pipeline up and running again".
I was in a fast-food restaurant when their credit card processor went down. They continued to take orders and if the customer didn't have cash the food was free. The cost of poor service and losing repeat customers was greater than the reduction in profit.
Now, this was a semi-independent franchise owned by a local family. Across the street, the chain big-box store had the same thing happen and they put a sign on the door saying "cash only".
Now, this was a semi-independent franchise owned by a local family. Across the street, the chain big-box store had the same thing happen and they put a sign on the door saying "cash only".
> if a store's computers go down they don't just let customers walk out the door with products and hope they get paid eventually?
This is exactly what we expect them to do in a crisis situation. This is also how Hollywood portrays ethical behavior. The store owner in Jericho (2006 TV series) is the first to come to mind, but there should be plenty of other examples.
And good deeds come back to us.
This is exactly what we expect them to do in a crisis situation. This is also how Hollywood portrays ethical behavior. The store owner in Jericho (2006 TV series) is the first to come to mind, but there should be plenty of other examples.
And good deeds come back to us.
The difference is that the Colonial Pipeline is seen as an important part of the US Critical National Infrastucture - without oil large parts of the economy and security system breakdown, people can die. It's protected by Homeland Security and the FBI too a much greater extent than your average store.
Ok, so the government steps in and says the pipeline operates without a billing system... now what?
Pipes are useless without something to flow through them, and no oil producer is going to fill the pipe without a system to get paid.
Pipes are useless without something to flow through them, and no oil producer is going to fill the pipe without a system to get paid.
The story was that hackers had closed the pipeline (or the owners had closed it because its systems become compromised).
If the pipeline is seen as a piece of critical infrastructure [1] - where its closure can damage other parts of the US - then if Colonial didn't tell the FBI, CISA, the Department of Transport (or whoever is in charge of that part of infrastructure) that the reason they were closing it was because they couldn't bill their customers rather than it being a safety issue, I imagine they are in trouble.
They were lucky in that it was only down for a short period, perhaps its OK - if it had been down long enough so ambulances stopped running and food and goods weren't delivered - then I'm sure the government would have told them to restart the pipeline and billing would have been sorted out some other way.
[1] https://en.wikipedia.org/wiki/Critical_infrastructure_protec...
If the pipeline is seen as a piece of critical infrastructure [1] - where its closure can damage other parts of the US - then if Colonial didn't tell the FBI, CISA, the Department of Transport (or whoever is in charge of that part of infrastructure) that the reason they were closing it was because they couldn't bill their customers rather than it being a safety issue, I imagine they are in trouble.
They were lucky in that it was only down for a short period, perhaps its OK - if it had been down long enough so ambulances stopped running and food and goods weren't delivered - then I'm sure the government would have told them to restart the pipeline and billing would have been sorted out some other way.
[1] https://en.wikipedia.org/wiki/Critical_infrastructure_protec...
This wasn't even a shitty take -- most of the developed world has government-owned utilities so the thought that the entire east coast gas infrastructure could go down just because some private company had their billing infrastructure locked up is not only laughable, it's morally outrageous. This situation is just begging for federal regulation, and that's the conversation we should be having -- should laws be passed requiring companies like Colonial to keep pumping gas (on their on dime) even when their billing infrastructure goes down? I think yes and I think most of the general public would as well.
Instead of having that discussion, we have no discussion because people like you have decided to flag this issue. This is the problem with HN.
Instead of having that discussion, we have no discussion because people like you have decided to flag this issue. This is the problem with HN.
I agree with you that the company can't make a decision to simply ship fuel for free. This is where the government should step in. You can't have this kind of wide-scale economic disruption occurring. Government could mandate that they resume shipments, have them bill manually as best they can, and maybe use tax dollars can plug any holes after the fact (and if necessary).
They should also be fined for letting their billing system get compromised.
They should also be fined for letting their billing system get compromised.
I think the point they’re really trying to make is that gas should be treated like a utility - you shouldn’t provide it solely based on profitability for a corporation. But you are right that the way things are now, it’s quite clear why they shut down, and that outrage at this particular company seems to really be outrage at the system which it operates in.
>Of course they shut down if they couldn't bill?
I know it's a commercial operation but there's strategic reasons to keep it up. The attention it got from the US is evidence of that. Plus, they could have just made an educated guess on billing and resolved it later. Even the US could have made cash flow happen if that was an issue. And anyone who tried to take advantage of that would likely get the US's attention. Not worth it long-term.
But all this was overshadowed by the risk of ransomware possibly attacking the industrial systems which is really why it was shut down.
I know it's a commercial operation but there's strategic reasons to keep it up. The attention it got from the US is evidence of that. Plus, they could have just made an educated guess on billing and resolved it later. Even the US could have made cash flow happen if that was an issue. And anyone who tried to take advantage of that would likely get the US's attention. Not worth it long-term.
But all this was overshadowed by the risk of ransomware possibly attacking the industrial systems which is really why it was shut down.
Regardless, the moral of the story is this:
To hack and shut down infrastructure does *not* require penetrating the actual infrastructure. Instead there are (likely far more vulnerable) "support systems" that if compromised will have the same effect. Why pick a hardened target when a soft one will do?
It's like arresting Al Capone. He didn't go down for all the violence, etc., instead it was tax evasion.
To hack and shut down infrastructure does *not* require penetrating the actual infrastructure. Instead there are (likely far more vulnerable) "support systems" that if compromised will have the same effect. Why pick a hardened target when a soft one will do?
It's like arresting Al Capone. He didn't go down for all the violence, etc., instead it was tax evasion.
Isn't the argument that it should be a utility and not a purely for-profit business? Would you expect your local water company to shut off supply if they have a similar bookkeeping problem?
People, and particularly software people I've noticed, seem to chronically underestimate the magnitude of B2B where you have a potential number of different vendors to settle with in the 6 digits, that changes over time according to vendor internal reorg, closings, openings, relocations, changes of payment and financial service outsourcing arrangement, etc.
That doesn't at all come for free. Entire departments and businesses revolve around making that complexity disappear. Gettting locked out of your infrastructure to do that is essentially paralysis of your business period. It isn't a case of "Just put some meters on the tap of the pipe" .
You have to be able to read them, you have to know to read them and when, you've got to know where the invoice goes, what terms were negotiated, etc. If you don't have access to that dataset, the only physical way for the human network to recreate that web is by people picking up phones and blowing up and slamming your point of contact which runs into the issue of having enough people on your end to handle things in the meantime, and your customers actually knowing who to call, and javing planned a procedure for a massive degradation in your info system, even if it means going back to pen and paper.
That doesn't at all come for free. Entire departments and businesses revolve around making that complexity disappear. Gettting locked out of your infrastructure to do that is essentially paralysis of your business period. It isn't a case of "Just put some meters on the tap of the pipe" .
You have to be able to read them, you have to know to read them and when, you've got to know where the invoice goes, what terms were negotiated, etc. If you don't have access to that dataset, the only physical way for the human network to recreate that web is by people picking up phones and blowing up and slamming your point of contact which runs into the issue of having enough people on your end to handle things in the meantime, and your customers actually knowing who to call, and javing planned a procedure for a massive degradation in your info system, even if it means going back to pen and paper.
The author here is misrepresenting his sources to spin a specific agenda point, in order to construct a false narrative. It's as fake as fake news gets.
From the author's own sources...
> Colonial’s corporate IT network and the process control network are connected and exchange information about how much fuel each supplier or distributor receives in order to bill them for it.
> ...they could change [data about the] flow rates, they could modify the data
> Although infecting Colonial Pipeline’s process control network would be disruptive, it isn’t the only concern. Colonial’s control system also connects to the control systems at tank farms that feed fuel into Colonial’s pipeline.... An attacker can potentially pass through Colonial’s control systems into the control systems of these farms.
I really hate advocacy journalism. It's such a fundamentally dishonest medium.
From the author's own sources...
> Colonial’s corporate IT network and the process control network are connected and exchange information about how much fuel each supplier or distributor receives in order to bill them for it.
> ...they could change [data about the] flow rates, they could modify the data
> Although infecting Colonial Pipeline’s process control network would be disruptive, it isn’t the only concern. Colonial’s control system also connects to the control systems at tank farms that feed fuel into Colonial’s pipeline.... An attacker can potentially pass through Colonial’s control systems into the control systems of these farms.
I really hate advocacy journalism. It's such a fundamentally dishonest medium.
IMHO, the USA has made the mistake of privatizing too much of its crucial infrastructure, ignoring that certain things are inherently monopolistic in nature and do not benefit at all from being in private hands.
Privatizing infrastructure is extremely positive if there's any chance for competition to arise - for instance, Italy has moved from a monopolistic state phone company to a well regulated and open free market, which has caused a boom of competing companies that has massively driven prices down. I now pay €27/mo for unlimited 1Gbps/300Mbps fiber and €7/mo for my phone plan with 50 GB of data, unlimited calls and unlimited SMS.
This is offset by the sad state of our highway system, which has been handed off to private companies a long time ago. There is obviously no way to create competition in highways (what are you going to do, build a cheaper highway next to the already existing one?), so the main company (API) has been neglecting mantaining infrastructures up to the point that a few bridges have collapsed, killing people and creating huge preventable disasters. Tolls are also crazy expensive, often amounting to half the cost of a road trip (and here petrol costs €1.50 per litre).
There's no incentive for a private company to invest a single cent as soon as it becomes a monopoly, it is something that has been known since the dawn of man.
Privatizing infrastructure is extremely positive if there's any chance for competition to arise - for instance, Italy has moved from a monopolistic state phone company to a well regulated and open free market, which has caused a boom of competing companies that has massively driven prices down. I now pay €27/mo for unlimited 1Gbps/300Mbps fiber and €7/mo for my phone plan with 50 GB of data, unlimited calls and unlimited SMS.
This is offset by the sad state of our highway system, which has been handed off to private companies a long time ago. There is obviously no way to create competition in highways (what are you going to do, build a cheaper highway next to the already existing one?), so the main company (API) has been neglecting mantaining infrastructures up to the point that a few bridges have collapsed, killing people and creating huge preventable disasters. Tolls are also crazy expensive, often amounting to half the cost of a road trip (and here petrol costs €1.50 per litre).
There's no incentive for a private company to invest a single cent as soon as it becomes a monopoly, it is something that has been known since the dawn of man.
Not just in the US. In Germany economic-liberal politicians keep pushing for privatized infrastructure all the time, arguing it would be better managed by a private company.
This goes as far as privatizing toll collection on public streets. Unsurprisingly this always leads to badly maintained infrastructure and higher costs in the end.
This goes as far as privatizing toll collection on public streets. Unsurprisingly this always leads to badly maintained infrastructure and higher costs in the end.
Competition is not only ineffective in monopolistic markets. Also oligopolic markets are very inefficient. In Italy's example the price you now pay for your mobile plan (7 €/month for 50 GB) was made possible by Iliad, a new ISP that broke the Italian oligopoly and forced all competitors to bring prices down (or better data caps up) ten fold in the span of a few months.
No, not tenfold. Iliad entered a competitive market, but my pre-iliad price was around 10 eur/month, albeit with smaller data allowance.
Iliad helped lowering prices by offering a lower end service.
Iliad helped lowering prices by offering a lower end service.
Australia has followed America in making that mistake too, sadly. Other countries as well I assume. The politicians that push this, seem to make sure it’s their cronies that are the ones that get to make billions off said privatisation, too.
On the other hand you have countries like Norway, where parts of the profits earned from oil goes into the "Oil Fund" (https://en.wikipedia.org/wiki/Government_Pension_Fund_of_Nor...), currently the "world’s largest sovereign wealth fund". If Norway had failure like the US, they could easily offset the costs while making things still work as usual for a while, as they are thinking ahead.
Is there good info on this somewhere?
Colonial Pipeline is a common carrier, "moving gasolines, kerosenes, home heating oils, diesel fuels and national defense fuels" (the US military runs on something which has the additives to be used either as JET-A or diesel fuel) "to shipper terminals in 12 states and the District of Columbia." Some pipelines switch from one product to another, usually with a slug of water in between for separation. There's valve switching, tank filling, and tank emptying going on. If you can't coordinate that, the wrong product ends up in the wrong place. If they lost the system that tracks what's where and where it's going, they have to shut down, or, as they seem to have done, dropped the pumping speed way down so they could operate the system manually.
It's not just one pipe that carries one product.
Colonial Pipeline is a common carrier, "moving gasolines, kerosenes, home heating oils, diesel fuels and national defense fuels" (the US military runs on something which has the additives to be used either as JET-A or diesel fuel) "to shipper terminals in 12 states and the District of Columbia." Some pipelines switch from one product to another, usually with a slug of water in between for separation. There's valve switching, tank filling, and tank emptying going on. If you can't coordinate that, the wrong product ends up in the wrong place. If they lost the system that tracks what's where and where it's going, they have to shut down, or, as they seem to have done, dropped the pumping speed way down so they could operate the system manually.
It's not just one pipe that carries one product.
How come that billing is basically ALWAYS the typical system that - no matter what - runs on Windows? SMEs software, big enterprise software and everything in between, you can have 99% of your core systems running on whatever operating system you want, or several of them, but the billing/payroll/accounting system will ALWAYS be Windows. Even if SAP has Linux/Unix versions.
I bet their billing system is custom and was developed in times when Windows was a better choice. And nobody was able (or even tried) to justify a switch since then.
At what point in history was that?
Unix(tm) and mainframes predate windows server by a far margin and was never seen as inferior to windows by the wider sysadmin community, there might have been a time when windows server had merit as the discount solutions for people who could not afford highly available Unix boxen, but it was never seen as superior or particularly well suited for critical infrastructure.
I think that's kind of whats in play here, few finance systems were designed as critical as well finance was that department that generated a few reports that nobody really read and rarely interacted with the core productive business to the point where flaws/bugs in finance systems could have an real impact until very recently so what you have is a bunch of aging discount/non-critical system that have been promoted to critical infrastructure nearly by accident.
The same goes for the desktop support infrastructure build around AD and SMB file shares, where systems that used to be auxiliary nice to have for clerical workers ended up as critical for the actual productive divisions without ever receiving any real hardening.
Unix(tm) and mainframes predate windows server by a far margin and was never seen as inferior to windows by the wider sysadmin community, there might have been a time when windows server had merit as the discount solutions for people who could not afford highly available Unix boxen, but it was never seen as superior or particularly well suited for critical infrastructure.
I think that's kind of whats in play here, few finance systems were designed as critical as well finance was that department that generated a few reports that nobody really read and rarely interacted with the core productive business to the point where flaws/bugs in finance systems could have an real impact until very recently so what you have is a bunch of aging discount/non-critical system that have been promoted to critical infrastructure nearly by accident.
The same goes for the desktop support infrastructure build around AD and SMB file shares, where systems that used to be auxiliary nice to have for clerical workers ended up as critical for the actual productive divisions without ever receiving any real hardening.
That was until 15-20 years ago.
*nix OSes may have been as far superior to Windows as you can imagine, but the development ecosystem certainly wasn't. There were Delphi, VB and other RAD tools for Windows, with tons of GUI and general purpose libraries and toolkits. Bun on *nix you basically had just GCC and vim/emacs. There were no GitHub, no npm, no AWS, nothing. If you'd compare TCO of an enterprise app plus necessary infrastructure, the Windows-flavored one would come a clear winner. Even mixed environments were too much of a hassle, due to poor interoperability.
*nix OSes may have been as far superior to Windows as you can imagine, but the development ecosystem certainly wasn't. There were Delphi, VB and other RAD tools for Windows, with tons of GUI and general purpose libraries and toolkits. Bun on *nix you basically had just GCC and vim/emacs. There were no GitHub, no npm, no AWS, nothing. If you'd compare TCO of an enterprise app plus necessary infrastructure, the Windows-flavored one would come a clear winner. Even mixed environments were too much of a hassle, due to poor interoperability.
Anecdotal, but working in retail in the 90s, the backend system was some *nix server, the wireless Telxon guns ran DOS 3.2, and there was a dedicated OS/2 Warp desktop that would perform reporting and aggregate data.
I am not sure what the registers ran, but they were IBM, all text-based.
Also, I remember my friend was a manager at AMC theatres at the time, and their system was entirely text-based. I remember watching him run end-of-day reports and you could see it zip up the files and hear the modem dial out and connect to corporate.
I am not sure what the registers ran, but they were IBM, all text-based.
Also, I remember my friend was a manager at AMC theatres at the time, and their system was entirely text-based. I remember watching him run end-of-day reports and you could see it zip up the files and hear the modem dial out and connect to corporate.
Dont make the mistake of thinking that the crap that was the pure gnu toolchain was in any way equal to what was available for those who could afford to run commercial unix(tm), it was an completely different world that Linux would not catch up to until the 00's mostly by absorbing frameworks and software from the world of commercial unix(tm) and from more recent developments.
I dont think i can recall any commercial Unix even shipping gcc back in the day.
I dont think i can recall any commercial Unix even shipping gcc back in the day.
Thick desktops were ruled by Windows so if you wanted a custom application it was Microsoft Visual Basic or Visual C++, Java as desktop apps didn't seem to catch on for the enterprise. Even if you tried to bring an application to the web back in the 90s you could still end up caught in the Microsoft ecosystem via ActiveX.
And desktops were never critical infrastructure were almost exclusively talking about servers here not disposable client devices as there is no such thing as an highly available desktop/laptop.
I have personally been involved in more java fat client then activeX based fake webapps, but im also not an wintel admin with an background in non-critical mid sized business applications.
I have personally been involved in more java fat client then activeX based fake webapps, but im also not an wintel admin with an background in non-critical mid sized business applications.
Depending on the version of Windows, it may not have even had native TCP/IP support by default.
Trumpet Winsock was a very important piece of software for a bit, from Windows 3 to 98.
Serial ports were much more popular back then.
Trumpet Winsock was a very important piece of software for a bit, from Windows 3 to 98.
Serial ports were much more popular back then.
Microsoft Visual Basic and Visual C++ had quite the market hold when a lot of this was built in the 90s
If you're going to have a billing system that you need to rely on for the functioning of your business. You're going to want to have ongoing support for that system, this is why so many enterprise options are Windows based. Especcially in earlier years, and financial suites, once implemented, are very rarely ever changed. If its still bringing in money, the higher-ups dont care.
[deleted]
The author here is misrepresenting his sources to spin a specific agenda point, in order to construct a false narrative. It's as fake as fake news gets.
From the author's own sources...
> Colonial’s corporate IT network and the process control network are connected and exchange information about how much fuel each supplier or distributor receives in order to bill them for it.
> ...they could change [data about the] flow rates, they could modify the data
> Although infecting Colonial Pipeline’s process control network would be disruptive, it isn’t the only concern. Colonial’s control system also connects to the control systems at tank farms that feed fuel into Colonial’s pipeline.... An attacker can potentially pass through Colonial’s control systems into the control systems of these farms.
I really hate advocacy journalism. It's such a fundamentally dishonest medium.
From the author's own sources...
> Colonial’s corporate IT network and the process control network are connected and exchange information about how much fuel each supplier or distributor receives in order to bill them for it.
> ...they could change [data about the] flow rates, they could modify the data
> Although infecting Colonial Pipeline’s process control network would be disruptive, it isn’t the only concern. Colonial’s control system also connects to the control systems at tank farms that feed fuel into Colonial’s pipeline.... An attacker can potentially pass through Colonial’s control systems into the control systems of these farms.
I really hate advocacy journalism. It's such a fundamentally dishonest medium.
If that is true, it's that an admission that you do not view fuel supplies as critical infrastructure?
Even so, some one has to pay, and the government wouldn't be keen on just handing out free fuel either.
If the power grid in your area loses its billing system should it just shut down?
What I don’t get is why they couldn’t gather the data manually to keep things running and sorted it out after the fact. They didn’t have any DR planning for something like this? Station people with clipboards and cameras at all meters. Measure the inputs and outputs as they occur. It seems there was just no real impetus to act responsibly here.
What I don’t get is why they couldn’t gather the data manually to keep things running and sorted it out after the fact. They didn’t have any DR planning for something like this? Station people with clipboards and cameras at all meters. Measure the inputs and outputs as they occur. It seems there was just no real impetus to act responsibly here.
Who do you hire to do that?
How quickly can you train them?
Where do they send the collected information?
How is the collected info processed and aggregated?
How much money is lost doing the above versus just shutting down?
I think their DR was fine. It is the HA that was not.
$5mil from net revenue of $400 mil is less than 2% of profits, and it has been operating for 80 years.
Honestly, I think the environmental damage the pipeline has caused is much worse.
How quickly can you train them?
Where do they send the collected information?
How is the collected info processed and aggregated?
How much money is lost doing the above versus just shutting down?
I think their DR was fine. It is the HA that was not.
$5mil from net revenue of $400 mil is less than 2% of profits, and it has been operating for 80 years.
Honestly, I think the environmental damage the pipeline has caused is much worse.
That's expensive. You have to have someone do all that paerwork, the paperwork has to exist (printing and doc management), and there needs to be a hard copy of the business agreement on a per customer basis.
I mean, I'm kinda old timey, so ot"s not infeasible from my point of view; but in today's hyper-optimized, bottom-line over-fault-tolerance world, I don't see that happen all that much.
Heck, I have trouble getting a straight answer out of the youngin's to "What is a filing system?"
I mean, I'm kinda old timey, so ot"s not infeasible from my point of view; but in today's hyper-optimized, bottom-line over-fault-tolerance world, I don't see that happen all that much.
Heck, I have trouble getting a straight answer out of the youngin's to "What is a filing system?"
Bill them the same this month as you billed them last month.
Adjust the bill next month, in the light of figures.
Adjust the bill next month, in the light of figures.
[deleted]
If ever there was a good candidate for a company to bailout to prevent damage to the wider economy, this is it.
On the flip side you are bailing out a company for neglecting to have a disciplined security practice and a failure to build and execute their disaster recovery plan.
Rather than bail out the company perhaps nationalizing it is the better recourse. It would certainly make other companies seriously reconsider how they invest in DR and security.
Rather than bail out the company perhaps nationalizing it is the better recourse. It would certainly make other companies seriously reconsider how they invest in DR and security.
Can we get the title fixed please? Even if it's misspelled at the source.
..., shut down ...
..., shut down ...
Both are valid, it’s journalistic headline abbreviation. It can be read as:
> [the] shutdown [was] to make sure
Or with your correction it would be read as:
> [it was] shut down to make sure
So it’s not incorrect in the source article, it is just using the noun shutdown, rather than the verb to shut down.
> [the] shutdown [was] to make sure
Or with your correction it would be read as:
> [it was] shut down to make sure
So it’s not incorrect in the source article, it is just using the noun shutdown, rather than the verb to shut down.
You are being way too generous. Your first expansion is a very big stretch.
Perhaps, though I’m not arguing that it was a good headline. My point was only that it is a valid headline, as long as you understand that shutdown is the noun.
> didn’t actually shut down the pipeline. It impacted the billing system at the Colonial Pipeline Co., which shut it down because they were worried about how they’d collect payments.
What? Did the author think that the ransomeware literally locked valves?
What? Did the author think that the ransomeware literally locked valves?
Please try reading the article again.
"the operational network that controls its pipelines and distributes fuel is separate from the corporate network and wasn’t infected"
The article thinks it matters whether the ransomware gets on the operational network, not your strawman.
"the operational network that controls its pipelines and distributes fuel is separate from the corporate network and wasn’t infected"
The article thinks it matters whether the ransomware gets on the operational network, not your strawman.
> Did the author think that the ransomeware literally locked valves?
In the early times of the pipeline shutdown this was exactly what was feared. Malware attacking, damaging, destroying or abusing industrial embedded devices to persist in to survive cleanups is nothing new. Stuxnet was just the tip of the iceberg.
In the early times of the pipeline shutdown this was exactly what was feared. Malware attacking, damaging, destroying or abusing industrial embedded devices to persist in to survive cleanups is nothing new. Stuxnet was just the tip of the iceberg.
Yes, in fact, if a store's computers go down they don't just let customers walk out the door with products and hope they get paid eventually?
As far as I understand colonial is basically a big gas station connected with a series of tubes. Look ma, the gas pump still works but it doesn't keep track of how much we pumped or charge us any more!
Of course they shut down if they couldn't bill?
It's being treated like some kind of greed when in reality they didn't want to give away millions of dollars of fuel they couldn't track or bill for.
The colonial pipeline was fine... except it couldn't do the thing it does which is sell gas to distributors, it could only give it away.
Sigh.
This isn't "fake news", but it is... shitty news? News which gives facts and a ridiculous analysis?
I'm still trying to find out what actually happened in Israel but god forbid a news source or commentator actually lay out the details of a complicated situation... (if you're not getting everybody pissed off about X, then you're not doing your job as a reporter?)