OptiFi Program Incident Report(medium.com)
medium.com
OptiFi Program Incident Report
https://medium.com/@OptiFi/optifi-program-incident-report-08-29-22-d8fe6d229bad
157 comments
As someone who just heard about FusionIO cards...how did the config change eat all the data in a manner that wasn't recoverable?
Perhaps they only discovered it after a while?
In the comment benreesman only said that enough data was deleted to cause 50-100M$ in damages. They might very well have managed to save half the data, but the lost half was already quite expensive?
In the comment benreesman only said that enough data was deleted to cause 50-100M$ in damages. They might very well have managed to save half the data, but the lost half was already quite expensive?
If MM is million-million then that is a lot of money.
Edit: thanks for replies! :)
Edit: thanks for replies! :)
MM is for mille-mille (French?), thousand-thousand, so 1000-1000 or 1-000-000.
See "How big is a billion?" from Numberphile:
* https://www.youtube.com/watch?v=C-52AI_ojyQ
* https://en.wikipedia.org/wiki/Long_and_short_scales
See "How big is a billion?" from Numberphile:
* https://www.youtube.com/watch?v=C-52AI_ojyQ
* https://en.wikipedia.org/wiki/Long_and_short_scales
In finance, MM is a shorthand for millions: https://www.orsurety.com/blog/is-it-m-for-thousand-and-mm-fo...
Weird to call it a shorthand when M means the same thing ;)
English wasn't always the lingua franca
But France has been on metric for a long time.
Yeah, but they counted money for even longer time
[deleted]
[deleted]
If MM were million-million, that would mean OP destroyed $50-100 trillion dollars. About the GDP of the entire world. So it probably doesn't mean million-million ;)
MM is million
Was it Cambridge Analytica that was going to pay 50MM for that data? ;)
No, it was features for a P(click|impression) model, which are surprisingly mundane: it is basically impossible to care much either way, let alone be outraged, if you look at the actual features used in these kinds of things and it’s not explicitly your field. It’s not a microphone listening on your smartphone!
You can even read about it in this paper which publicly described the system contemporary to that event: http://www.quinonero.net/Publications/predicting-clicks-face...
If you read it and are still steamed up, well, then you hold that opinion honestly, but I suspect most would be like, “cool story bro”.
You can even read about it in this paper which publicly described the system contemporary to that event: http://www.quinonero.net/Publications/predicting-clicks-face...
If you read it and are still steamed up, well, then you hold that opinion honestly, but I suspect most would be like, “cool story bro”.
I'll just file it under "best minds of our generation are working on making people click 3% more often on ads".
Bernoulli distributions are everywhere, and as a result binary classifiers are everywhere. COVID tests (any binary medical diagnostic), fraud/no-fraud credit-card processing decisions, loan decisions, spam filtering, “toxic” tweet management and other sentiment analysis, it just goes on forever.
All of these things benefit from advances in binary classifiers, and have therefore been massively subsidized by click prediction R&D. Maybe those benefits outweighs the costs, maybe they don’t.
We have GPUs because shoot-em-up games were “the big thing” in the 90s. Net win? Personal preference. But entertainment if one form or another often pushes advances in computing. The BOPR stuff from the linked paper originated on XBox Live matchmaking.
It’s perfectly fair to be like: I don’t like the ad-funded internet. There are tons of plausible reasons why someone wouldn’t like the ad-funded Internet, but most reasonable objections to it are of the form: “my internet experience is worse than it would be without ad funding”, past that you rapidly wander into “other people’s preferences are inferior to mine”.
All of these things benefit from advances in binary classifiers, and have therefore been massively subsidized by click prediction R&D. Maybe those benefits outweighs the costs, maybe they don’t.
We have GPUs because shoot-em-up games were “the big thing” in the 90s. Net win? Personal preference. But entertainment if one form or another often pushes advances in computing. The BOPR stuff from the linked paper originated on XBox Live matchmaking.
It’s perfectly fair to be like: I don’t like the ad-funded internet. There are tons of plausible reasons why someone wouldn’t like the ad-funded Internet, but most reasonable objections to it are of the form: “my internet experience is worse than it would be without ad funding”, past that you rapidly wander into “other people’s preferences are inferior to mine”.
Another example: we are finally getting rid of internal combustion in cars, because the phone industry paid for the battery research.
Thanks that's interesting! Definitely not outraged, was just poking a bit for fun. 50MM is an insane amount of money/value that would bankrupt most companies and have a noticable impact on many industries. Funny that for Facebook it was probably just a small hiccup on the ad revenue gravy train.
> 50MM is an insane amount of money/value that would bankrupt most companies [...]
Well, it would probably bankrupt most companies, but it wouldn't bankrupt most people's employer.
The average employee works at a larger than average company. Similar for the average customer or average investor.
That's just because big companies have more employees / customers / investors to contribute to that kind of average.
Well, it would probably bankrupt most companies, but it wouldn't bankrupt most people's employer.
The average employee works at a larger than average company. Similar for the average customer or average investor.
That's just because big companies have more employees / customers / investors to contribute to that kind of average.
I've watched crypto for years with interest, I've worked in the field, speculated in it with my savings, used it for international transfers, spent it, etc. That is to say I'm not some armchair critic.
I find myself lately viewing crypto as mostly a bad idea, or a bad implementation of an idea with potential, or even outright as a Ponzi scheme.
These smart contacts are really a bad idea. We have contracts in life with lawyers, these tend to work out pretty well because we can use our human intelligence to arbitrate them. The idea of code as law is really dumb. I say that as a programmer who works on code all day. Computers are dumb like your calculator. They do exactly what you tell them to. Which sounds great until you realize how hard it is to describe what you want in a way the computer can understand. That's why I make a good salary and people think my job is difficult. The reality is computers often do what you tell them, but it's not what you wanted. Rather you have the instructions wrong or you made a mistake. That's why software is riddled with bugs. Avoiding that is very, very, very difficult. Nobody gets it right. These crypto finance bros will definately not get it right. And when they get it wrong, there's no human, no court to appeal to. How's that an improvement?
I find myself lately viewing crypto as mostly a bad idea, or a bad implementation of an idea with potential, or even outright as a Ponzi scheme.
These smart contacts are really a bad idea. We have contracts in life with lawyers, these tend to work out pretty well because we can use our human intelligence to arbitrate them. The idea of code as law is really dumb. I say that as a programmer who works on code all day. Computers are dumb like your calculator. They do exactly what you tell them to. Which sounds great until you realize how hard it is to describe what you want in a way the computer can understand. That's why I make a good salary and people think my job is difficult. The reality is computers often do what you tell them, but it's not what you wanted. Rather you have the instructions wrong or you made a mistake. That's why software is riddled with bugs. Avoiding that is very, very, very difficult. Nobody gets it right. These crypto finance bros will definately not get it right. And when they get it wrong, there's no human, no court to appeal to. How's that an improvement?
For all its flaws (and there are countless) the one thing about the show cryptocurrency space that stands out to me as a programmer is that programming errors can be suddenly very costly (granted, in this case it was more of a DevOps blunder).
Being able to very easily put a price tag on sloppy programming is intriguing to me.
Being able to very easily put a price tag on sloppy programming is intriguing to me.
You (and everyone really) should probably read some more comp.risks http://catless.ncl.ac.uk/Risks/
Certain fields have always had a high cost of programming error, including a cost in human life. It's just that cryptocurrency combines this with a first-to-market rush that's somehow still going on, encouraging a rush to error.
Certain fields have always had a high cost of programming error, including a cost in human life. It's just that cryptocurrency combines this with a first-to-market rush that's somehow still going on, encouraging a rush to error.
I don't disagree with that there have always been niche areas where there is a high cost to programming errors (like space or medical).
But with crypto, the effect is much more direct. The programmer is handling money much more directly and if something goes wrong, they are much more directly affected and not being insulated from the effects.
But with crypto, the effect is much more direct. The programmer is handling money much more directly and if something goes wrong, they are much more directly affected and not being insulated from the effects.
At the first proper job I had we would measure our service outages in dollars per second. It was always in the thousands, and we were just a rather ordinary financial OLTP service.
First to market is only part of the problem. Another self inflicted problem is that execution of those "smart" "contracts" cost tokens (so essentially money). This incentivizes developers to write the shortest possible code, without any "fluff" like tests or additional checks or more verbose style. And the immutability problem also increases severity of the problem - you either deploy immutable "contract" signifying that it should be respected because it can't be changed later for malicious purpose, or you deploy modifiable "contract" and that is not good for the reputation of the company.
Immutable code is a poor replacement for trust in a company.
It shifts the trust from the company as it is now to the same company (and their capabilities) sometimes in the past. Preferably the company is operated completely anonymously.
It seems like the crypto community would do anything to avoid the legal/regulatory system and it's established processes of operation requirements, insurance and liabilities.
Doing that they regularly fail at performing the most basics of basic financial duties like not getting hacked and not throwing the keys to the kingdom.
I've yet to hear of a bank loosing funds and getting away with telling "tough luck" to their customers, but we've witnessed many crypto "banks" doing just that.
It seems like the crypto community would do anything to avoid the legal/regulatory system and it's established processes of operation requirements, insurance and liabilities.
Doing that they regularly fail at performing the most basics of basic financial duties like not getting hacked and not throwing the keys to the kingdom.
I've yet to hear of a bank loosing funds and getting away with telling "tough luck" to their customers, but we've witnessed many crypto "banks" doing just that.
The closest I can think of is the TSB migration failure: https://www.theguardian.com/business/2019/nov/19/tsb-it-melt...
That also involved failing to have a viable rollback plan. It affected so many people so badly that questions were asked in Parliament; TSB were down for a week.
Now with cryptocurrency we've disintermediated the bank to produce a much better solution: your contract can be down forever and there's no Parliament to ask questions in.
That also involved failing to have a viable rollback plan. It affected so many people so badly that questions were asked in Parliament; TSB were down for a week.
Now with cryptocurrency we've disintermediated the bank to produce a much better solution: your contract can be down forever and there's no Parliament to ask questions in.
When the price tag is “everything you own”, how is this an improvement? Who wants to give over their financial life to a computer program?
This aspect of crypto is a 1960s paranoid computer fear come to life, and somehow it’s often presented as an improvement over the existing system of human checks and balances.
This aspect of crypto is a 1960s paranoid computer fear come to life, and somehow it’s often presented as an improvement over the existing system of human checks and balances.
> Who wants to give over their financial life to a computer program?
Where do you think your income and bank balances are tracked and stored? On pieces of paper?
Where do you think your income and bank balances are tracked and stored? On pieces of paper?
This is missing the point to a degree that makes me think you're being intentionally obtuse, but maybe you're just ignorant so I'll bite. Banking computer errors can easily be rectified by humans, banks are regulated, your funds are at least partially guaranteed by the government (depending on where you live). The degree to which you're trusting computer programs with your finances is orders of magnitude less than with cryptocurrency where it's possible to lose any amount $ of asset value in an instant with absolutely no means of recourse.
Your bank analogy is silly and nowhere near analogous.
Your bank analogy is silly and nowhere near analogous.
CitiBank can't get the money back they accidentally transferred to another company.
https://www.bloomberg.com/news/articles/2022-08-15/citi-sues...
You can say that's an edge case today but I and OP are saying, the future will look more like crypto looks today. Not a bright future.
https://www.bloomberg.com/news/articles/2022-08-15/citi-sues...
You can say that's an edge case today but I and OP are saying, the future will look more like crypto looks today. Not a bright future.
>CitiBank can't get the money back they accidentally transferred to another company.
Given it was an accidental early repayment of a loan, this isn't quite the slam dunk you think it is. If they had paid a company they didn't owe money to, they could get the money back through the courts.
Crypto is meant to evade those courts.
Given it was an accidental early repayment of a loan, this isn't quite the slam dunk you think it is. If they had paid a company they didn't owe money to, they could get the money back through the courts.
Crypto is meant to evade those courts.
The reason was not because of a repayment but because you wouldn't expect a respectable bank like CitiBank to do such a mistake.
> “To believe that Citibank, one of the most sophisticated financial institutions in the world, had made a mistake that had never happened before, to the tune of nearly $1 billion, would have been borderline irrational,” he wrote.
https://www.nytimes.com/2021/02/16/business/citibank-revlon-...
That's as close to "code is law" and other nonsense of the crypto libertarians as you can get.
> “To believe that Citibank, one of the most sophisticated financial institutions in the world, had made a mistake that had never happened before, to the tune of nearly $1 billion, would have been borderline irrational,” he wrote.
https://www.nytimes.com/2021/02/16/business/citibank-revlon-...
That's as close to "code is law" and other nonsense of the crypto libertarians as you can get.
Not really. Right in the paragraph above that one:
> Recipients of cash wired in error are typically required to return it.
> Recipients of cash wired in error are typically required to return it.
> I and OP are saying, the future will look more like crypto looks today. Not a bright future.
I don’t agree. I trust that loopholes like that will be slowly rectified with legislation if not present today.
In fact, it’s likely that crypto will (problematically?) be heading the same way. There was a recent case of a crypto buff who found a bug in some project and made off with a few $Million and I think the courts said he could be arrested and expected to return it, just as if he made off with cash. Importantly, they basically said “blockchain isn’t the source of truth to the courts” which was the guys defense. IMO a bright future for people, but not for a crypto venture.
I don’t agree. I trust that loopholes like that will be slowly rectified with legislation if not present today.
In fact, it’s likely that crypto will (problematically?) be heading the same way. There was a recent case of a crypto buff who found a bug in some project and made off with a few $Million and I think the courts said he could be arrested and expected to return it, just as if he made off with cash. Importantly, they basically said “blockchain isn’t the source of truth to the courts” which was the guys defense. IMO a bright future for people, but not for a crypto venture.
> he could be arrested and expected to return it, just as if he made off with cash.
You have cash in your house and someone breaks in to steal everything. The insurance will _maybe_ cover your loss given that you secured it with basic security. _Maybe_ the police will investigate and arrest the burglar.
But even in this situation, you were better off putting your savings in any bank account where any fraudulent transaction can be reverted with a button.
Now, you have your crypto wallet. It gets emptied by some random bot. Well, you are as fucked than with your cash, except that nobody will cover your loss and nobody will investigate your case since the burglar is probably from another country.
You have cash in your house and someone breaks in to steal everything. The insurance will _maybe_ cover your loss given that you secured it with basic security. _Maybe_ the police will investigate and arrest the burglar.
But even in this situation, you were better off putting your savings in any bank account where any fraudulent transaction can be reverted with a button.
Now, you have your crypto wallet. It gets emptied by some random bot. Well, you are as fucked than with your cash, except that nobody will cover your loss and nobody will investigate your case since the burglar is probably from another country.
By cash I meant fiat. But yea crypto adds a lot of risk to storing your monies.
You can insure the crypto you have. It’s probs expensive to the point of being not worth it.
People investigate crypto hacks though. And if the perpetrators are in a jurisdiction that you have some legal availability to you can totally use legal means. Basically any western nation will allow such a suit.
https://www.coindesk.com/policy/2021/12/22/teenage-suspect-i...
You can insure the crypto you have. It’s probs expensive to the point of being not worth it.
People investigate crypto hacks though. And if the perpetrators are in a jurisdiction that you have some legal availability to you can totally use legal means. Basically any western nation will allow such a suit.
https://www.coindesk.com/policy/2021/12/22/teenage-suspect-i...
> When the price tag is “everything you own”, how is this an improvement? Who wants to give over their financial life to a computer program?
You can strike out "financial" and that's already the realty we're living in. We got lucky that Y2K was not an issue today. We wouldn't probably be able to fix enough code and nowadays much more is under direct computer control than back then.
Nobody "wants" that but it happens gradually. With crypto, it didn't happen organically but crypto blasted onto the scene from the side of complete digitization. Looking at that, it's easy to say that nobody would want that but while you're looking this way, traditional finance is creeping towards complete digitization as well, just behind your back.
> This aspect of crypto is a 1960s paranoid computer fear come to life, and somehow it’s often presented as an improvement over the existing system of human checks and balances.
I'm not saying that crypto is an improvement as it is now. Nor that it will ever be (it might but I have my doubts). But what you're missing is that crypto is complete wild west, like traditional finance was maybe in 1900. The whole history of finance is a sequence of fuckups and laws and regulations that were imposed to prevent similar fuckups to occur and we still got the 2007 financial crisis, after 150 years of improvements.
I can try to stay as far away from crypto as possible but as I said in my previous post, there are aspects that are interesting even if most of it is completely nuts.
You can strike out "financial" and that's already the realty we're living in. We got lucky that Y2K was not an issue today. We wouldn't probably be able to fix enough code and nowadays much more is under direct computer control than back then.
Nobody "wants" that but it happens gradually. With crypto, it didn't happen organically but crypto blasted onto the scene from the side of complete digitization. Looking at that, it's easy to say that nobody would want that but while you're looking this way, traditional finance is creeping towards complete digitization as well, just behind your back.
> This aspect of crypto is a 1960s paranoid computer fear come to life, and somehow it’s often presented as an improvement over the existing system of human checks and balances.
I'm not saying that crypto is an improvement as it is now. Nor that it will ever be (it might but I have my doubts). But what you're missing is that crypto is complete wild west, like traditional finance was maybe in 1900. The whole history of finance is a sequence of fuckups and laws and regulations that were imposed to prevent similar fuckups to occur and we still got the 2007 financial crisis, after 150 years of improvements.
I can try to stay as far away from crypto as possible but as I said in my previous post, there are aspects that are interesting even if most of it is completely nuts.
The level of self-perceived vs. actual competence in the crypto space never ceases to amaze.
A mea culpa of “The one thing we purport to be good at we actually have literally no understanding of and when shit doesn’t work we just run it a few times with different arguments.” My god.
A mea culpa of “The one thing we purport to be good at we actually have literally no understanding of and when shit doesn’t work we just run it a few times with different arguments.” My god.
Please send $100 each time you have a bug in your code, to following ETH wallet; 0x17C489878E89136cd399D162ECCed25924749822
You're welcome.
You're welcome.
When I first found about cryptocurrencies it struck me as it was the first time in the history where data on my computer were directly worth something. Which suddenly made security worth something.
Commands like that really need a confirmation prompt and a command line switch to override like --do-as-i-say (long form only). Good example of developers being put in an end user's shoes, I hope they learn from that mistake and update their programs.
An example of such a command came up on Linus Tech Tips (LTT), from apt (IIRC) on Pop_OS!. Basically (paraphrasing) "this will destroy your system and render it useless; type 'I'm an idiot and wish to proceed'". Our eponymous youtuber went right on, typed the phrase and blitzed the system, and then moaned about how the system shouldn't have let him break it ... I suspect such prompts only work for cautious people, who possibly don't need the prompts as they'll actually read the scary warnings.
In his defense, it was literally the first package he tried to install, which was in the official repositories but horribly broken, so he had no real way to tell whether this was the normal flow and linux just made it really irritating to install packages.
And the more relevant error message was buried in the scrollback, IIRC.
That guy is a savvy YouTuber, man. He did that because the resulting content has high memetic replication properties.
I find the prompt for dangerous ops useful. GitHub will ask you to repeat the name of the repo before you destructive actions, Terraform will ask you to say yes to the prompt. These are all good things.
I find the prompt for dangerous ops useful. GitHub will ask you to repeat the name of the repo before you destructive actions, Terraform will ask you to say yes to the prompt. These are all good things.
It's a solid practice in UX design. Physically having to type or copy/paste it in really highlights the action for the end user. There's only so much you can do to stop people setting their own house on fire but something like that puts the onus on them and fairly places the blame where it belongs.
UX is not about putting the blame on someone, but to prevent errors. For this reason confirmations are not really the best practice. Sure, they reduce the number of mistakes by some margin, but they never get it 100%.
The better alternative will be instead of asking for confirmation to ask to explain the intent by picking one of the options or writing it down. Ideally, an alternative must be suggested. This will avoid automatic reaction, because there's no one clear path to the goal.
The better alternative will be instead of asking for confirmation to ask to explain the intent by picking one of the options or writing it down. Ideally, an alternative must be suggested. This will avoid automatic reaction, because there's no one clear path to the goal.
The problems start when you know you want to light a house on fire, but you pick the wrong house.
It's almost never the case that blaming the user is actually going to help nor that adding more eyeballs will prevent people from making mistakes. If it's routine, we'll apply it to the wrong entity. If it's not routine, we'll not understand all the implications of our actions.
It's almost never the case that blaming the user is actually going to help nor that adding more eyeballs will prevent people from making mistakes. If it's routine, we'll apply it to the wrong entity. If it's not routine, we'll not understand all the implications of our actions.
Some actions have to be irreversible by design (think: emptying the trash to free up space on your drive, or deleting sensible user data). At some point, someone has to greenlight that action, and the best you can do is trying to ensure the user is aware of what they're about to do – and you have to trust that they're using their brain for once. You know, that thing in your head which distinguishes you from that thing sitting in the metal box under your desk... If it were possible to automate that decision in a flawless, 100% safe and correct way, there'd be no need for a human to press the button.
Life definitely has one-way gates, but tech can avoid them a lot more than we do today. Soft deletes and backups give a time-bound undo period to user actions, and so freeing up space on my drive isn't irreversible.
Also, to stretch the analogy I used further: one way to avoid lighting one's own house on fire by mistake is to not make the things you light on fire look like houses. In the case above, and in many other cases (like the big Atlassian outage earlier this year) the problem wasn't so much that the user was deliberately deleting important stuff, it's that they couldn't tell the difference between the class of unimportant things they thought they were deleting from and the class of important things they would stop and think long and hard about before deleting.
Also, to stretch the analogy I used further: one way to avoid lighting one's own house on fire by mistake is to not make the things you light on fire look like houses. In the case above, and in many other cases (like the big Atlassian outage earlier this year) the problem wasn't so much that the user was deliberately deleting important stuff, it's that they couldn't tell the difference between the class of unimportant things they thought they were deleting from and the class of important things they would stop and think long and hard about before deleting.
Yet there was some popular HN post a few months ago about someone complaining that GitHub didn't do enough to stop them...
It's a person who gets paid by views. They did this for the attention.
That's a technical solution to a psychological and organizational problem: when there should be no CLI in production, people still use it and get used to various errors and confirmations so much, that they ignore the signs of a catastrophe. Technical solutions never work in such cases.
And then there's a regulatory problem: investors trust their money to businesses which have not earned that trust, because of whatever magical thinking that exists on this market. At least the company seems to be able to return the money, but will it be sanctioned for this failure? There should be a regulatory incentive to do better next time.
And then there's a regulatory problem: investors trust their money to businesses which have not earned that trust, because of whatever magical thinking that exists on this market. At least the company seems to be able to return the money, but will it be sanctioned for this failure? There should be a regulatory incentive to do better next time.
> no CLI in production
It seems like this was in the "deploy to production" stage, and there has to be some mechanism for doing that.
(It also seems unnecessarily complicated; I'd appreciate a plaintext explainer of what actually went wrong)
It seems like this was in the "deploy to production" stage, and there has to be some mechanism for doing that.
(It also seems unnecessarily complicated; I'd appreciate a plaintext explainer of what actually went wrong)
Normally such deployment mechanism is one standard command that picks up a tested code, enriches it with environment configuration and rolls it out. This command rarely changes and should be the same as deployment to pre-production or staging. This command is usually executed from a CI tool via a button click and requires only user credentials as an input.
Any changes to production must be tested before release. They did not do it.
Any changes to production must be tested before release. They did not do it.
I don't know about you folks but I can do stupid stuff also with confirmation. Maybe splitting the damaging actions in some substeps, so instead of a big bang you have a slow death?
You are right, something like closing a program shouldn't be valid immediately, the program should be frozen for a day with the ability to cancel the closing at any time until the day is over.
Also, for me when I make big mistakes like this I think I’m in a different environment or something. It’s not that I don’t realise what command I’m running, it’s that I miss the context.
So I would just confirm but not realise what I was confirming.
So I would just confirm but not realise what I was confirming.
Sun has a good warning banner on their ZFS appliances. It made you thing twice before proceeding. Shown here: https://news.ycombinator.com/item?id=30195007
Command line switches like that won't, IMO and IME, make developers think "wow, I'm so glad this exists to guard against footguns". Rather they'll just set an alias to get around it. Likewise confirmation prompts albeit with a little more resitance. Once a repeat user is frustrated enough they'll bust out `expect` or similar, angry at the original command for mollycoddling them because they know what they're doing and don't make daft mistakes, damn it.
If you alias a —do-as-i-say switch to a command without wrapping into a script to make sanity checks of your own, IMO that’s negligence. The name of the flag should make you think thrice about invoking it without safety checks.
It was the same with safety overrides on hydraulic presses
People would tape one button on, then wonder why it would chop a finger sometimes
Yes it is annoying. Yes, there may be better choices. But it is there for a reason
People would tape one button on, then wonder why it would chop a finger sometimes
Yes it is annoying. Yes, there may be better choices. But it is there for a reason
I don't disagree. I think negligence happens.
I have not understood most of that article (optifi, mainnet, solana, PDAs, deployer address, program, tvl, ...what?) but what I get from it is that people who run crypto-anything make costly mistakes.
Funny that people still ask me "hey, why do you trust those evil banks? you should do all in crypto instead!" and then laugh at me when I tell them I trust banks more than some random dev on the internet...
Funny that people still ask me "hey, why do you trust those evil banks? you should do all in crypto instead!" and then laugh at me when I tell them I trust banks more than some random dev on the internet...
This is one of the biggest flaws in crypto. Small errors can erode hundreds of millions of value.
That's when most crypto companies right now are very small. Can you imagine the chaos if crypto were to actually become big and dev count were to grow to 1k+ people spread across multiple offices?
There's definite efficiency gains with crypto (a dex like Uniswap can do massive volume with very few developers for instance), but there needs to be a way to limit the cost of human errors.
That's when most crypto companies right now are very small. Can you imagine the chaos if crypto were to actually become big and dev count were to grow to 1k+ people spread across multiple offices?
There's definite efficiency gains with crypto (a dex like Uniswap can do massive volume with very few developers for instance), but there needs to be a way to limit the cost of human errors.
> This is one of the biggest flaws in crypto. Small errors can erode hundreds of millions of value.
That's reality.
Same thing would have happened if they had put all that cash on a boat, and accidentally sunk it.
That's reality.
Same thing would have happened if they had put all that cash on a boat, and accidentally sunk it.
Sure - but a lot of systems have a safety net of test environments and change control, and a safety net as they can revert changes, and a safety net as they can restore backups, and a safety net as they can ask counterparties nicely to help undo things, and a safety net through the legal system, and a safety net of insurance.
The normal banking industry is operating chainsaws very cautiously, with a lot of safety equipment and training. The cryptocurrency industry may be operating the same chainsaws, but they're trying to juggle them naked, on a floor slick with the blood of their peers.
The normal banking industry is operating chainsaws very cautiously, with a lot of safety equipment and training. The cryptocurrency industry may be operating the same chainsaws, but they're trying to juggle them naked, on a floor slick with the blood of their peers.
There's no reasons those safety nets can't exist here. Test nets, and insurance can take on those resposnibilities with the same effect.
> Same thing would have happened if they had put all that cash on a boat, and accidentally sunk it.
Are you saying that it is impossible to get valuables out of a boat that sunk?
"Treasure trove of gold and jewels recovered from a 366-year-old shipwreck in the Bahamas"
https://www.livescience.com/bahamas-shipwreck-gold-jewels
Are you saying that it is impossible to get valuables out of a boat that sunk?
"Treasure trove of gold and jewels recovered from a 366-year-old shipwreck in the Bahamas"
https://www.livescience.com/bahamas-shipwreck-gold-jewels
I'm saying cash dissolves in water.
If you'd prefer you can throw your gold bullion into a volcano, or perhaps mix it in with radioactive fuel.
If you'd prefer you can throw your gold bullion into a volcano, or perhaps mix it in with radioactive fuel.
This is a deeply stupid analogy, because as everyone is pointing out .. why would you do that? Why would you choose to build a system in which this is a risk, when it's not actually imposed by the laws of nature?
But there were a few instances in WW2 where large amounts of value had to be transferred by boat, and in at least one case sunk by enemy action. HMS Edinburgh: https://www.warhistoryonline.com/instant-articles/hms-edinbu...
But there were a few instances in WW2 where large amounts of value had to be transferred by boat, and in at least one case sunk by enemy action. HMS Edinburgh: https://www.warhistoryonline.com/instant-articles/hms-edinbu...
Sure. But there is no reason to put all your cash on a boat, so nobody does it (or it's because the risks outweighs the benefits).
Cash, being paper, would probably float.... Sadly for oceanic finance transfers, it's mostly gold, and that stuff sinks fast.
But we moved away from moving currency in sinkable boats hundreds of years ago. Error in plain old finance today can often be fixed. Do we need to regress back to antiquity?
> This is one of the biggest flaws in crypto. Small errors can erode hundreds of millions of value.
This is hardly exclusive to crypto.
> Can you imagine the chaos if crypto were to actually become big and dev count were to grow to 1k+ people spread across multiple offices?
Yeah I can already imagine in other places, just ask Santander Bank. [0]
[0] https://www.theguardian.com/business/2021/dec/30/santander-b...
This is hardly exclusive to crypto.
> Can you imagine the chaos if crypto were to actually become big and dev count were to grow to 1k+ people spread across multiple offices?
Yeah I can already imagine in other places, just ask Santander Bank. [0]
[0] https://www.theguardian.com/business/2021/dec/30/santander-b...
> Yeah I can already imagine in other places, just ask Santander Bank. [0]
Per the article, they were very likely to recover most of that money (and were legally entitled to recover all of it, but were unwilling to do so for image reasons). So exactly the opposite of what happens with DeFi.
Per the article, they were very likely to recover most of that money (and were legally entitled to recover all of it, but were unwilling to do so for image reasons). So exactly the opposite of what happens with DeFi.
It's not exactly unique to crypto... in adtech, a configuration mistake on a web server can cost 6-7 figures as ads are being paid for, but take people nowhere.
Sure, but the company managing the ad network can choose to be gentle, and refund the money. With these sorts of crypto mistakes, the money is just gone, because humans are -- as a desired feature of the system -- unable to correct these sorts of mistakes.
From the article:
Lessoned we learned harshly EVERY DEPLOYMENT NEEDS A RIGOROUS PROCESS AND SINGLE POINT FAILURE CAN BE AVOIDED. PLEASE DON’T RUSH LIKE WHAT WE DID, ESPECIALLY FOR DEFI PROJECTS.
Best wishes to the company and individuals involved for recovery of assets and success going forward.
Lessoned we learned harshly EVERY DEPLOYMENT NEEDS A RIGOROUS PROCESS AND SINGLE POINT FAILURE CAN BE AVOIDED. PLEASE DON’T RUSH LIKE WHAT WE DID, ESPECIALLY FOR DEFI PROJECTS.
Best wishes to the company and individuals involved for recovery of assets and success going forward.
Is there a good reason that the CLI is able to close a contract with money in it?
The money within the contract wasn't the problem, that was transferred out as part of closing the program.
However there were other assets on the chain that are locked using the program which are now inaccessible because its inoperable.
Consider it like an operation that closes and deletes your user-account but returns you an archive of your home directory when it's complete. However all files that weren't in your home directory that were encrypted with your local user key now can't be read.
It's likely infeasible to predicate program deletion on the program no longer being referenced in other programs/contracts.
Disclaimer: I'm not a Solana expert (or a crypto advocate for that matter)
However there were other assets on the chain that are locked using the program which are now inaccessible because its inoperable.
Consider it like an operation that closes and deletes your user-account but returns you an archive of your home directory when it's complete. However all files that weren't in your home directory that were encrypted with your local user key now can't be read.
It's likely infeasible to predicate program deletion on the program no longer being referenced in other programs/contracts.
Disclaimer: I'm not a Solana expert (or a crypto advocate for that matter)
This is why coins/tokens/contracts shouldn't have the code be updatable when they contain money...
If you want something where some developer can change the rules anytime, then just pay the money direct to the developer, and hope he pays you back one day.
If you want something where some developer can change the rules anytime, then just pay the money direct to the developer, and hope he pays you back one day.
Man, if only there was a central authority to appeal those mistakes.
It’s Solana, so there kinda is. It’s pretty much centralized.
Why is this called DeFi? If a dev can nuke an entire app (intentionally or accidentally) it's not decentralized.
Contrary to the HN zeitgeist, I have a lot of excitment for decentralization medium- to long term. But it seems almost everything that's happening in the DeFi space today is about as decentralized as a traditional client server app with mysql.
Contrary to the HN zeitgeist, I have a lot of excitment for decentralization medium- to long term. But it seems almost everything that's happening in the DeFi space today is about as decentralized as a traditional client server app with mysql.
Multiple DeFi platforms exist. Some are mostly decentralized, others only use DeFi as a marketing buzzword. Some of these are actually centralized, but it doesn't mean all of them are.
In agreement with GP, Solana and other very centralized networks should have to earn their way into the term DeFi. It's hard to quantify but a network that can be "paused" by few devs and relying on the existence of a registered company clearly don't fit this term.
Yeah. I don't get why bother at all with the blockchain when a simple dB would be even superior, then they can prototype away and migrate to decentralized later, if anyone even wants that. Decentralization shares the "weakest link" property with cyber security: you don't get the benefits of you only do half of it.
Because when you figure out Solana is a joke, you can take your millions off of their chain and onto Ethereum.
I need an explanation for this because the only arguments I have seen are Reddit memes that disregard the centralisation of mining pools.
This sort of thing is not exclusive to Solana. Ethereum will also join Solana and be another highly centralized PoS network, so I expect more of the same centralization points to apply to Ethereum and many other chains.
As for this issue, just look no further than the DAO hack and see if that has taught crypto anything about so-called decentralization; clearly it hasn't.
As for this issue, just look no further than the DAO hack and see if that has taught crypto anything about so-called decentralization; clearly it hasn't.
Well, that worked for Vitalik and the DAO.
> Strictly execute *peer-surveillance approach*that requires at least 3 peers to engage in the deployment process
> - We will adopt a peer-surveillance approach which requires at least 3 peers to engage in the deployment process. They have the responsibility to remind the main deployer of any potential risk, and make sure each step complies with the deployment guides and norms.
> - In case anything abnormal happens during the deployment process, such as bad network status or insufficient deployment fee, we should calm down and have a discussion with peers to make sure each operation is safe. Meanwhile, we should mark down every command line and returned message for further reference.
I wonder if that’s enough stack of Swiss Cheese to prevent such an accident from happening again. Hopefully they expand on the “we should calm down” to not be limited to saying “calm down please”. Calming down is quite difficult when you’re in the throes of something, and proper procedures tend to be put aside for the sake of pressure relief.
I wish them swift recovery from their predicament.
> - We will adopt a peer-surveillance approach which requires at least 3 peers to engage in the deployment process. They have the responsibility to remind the main deployer of any potential risk, and make sure each step complies with the deployment guides and norms.
> - In case anything abnormal happens during the deployment process, such as bad network status or insufficient deployment fee, we should calm down and have a discussion with peers to make sure each operation is safe. Meanwhile, we should mark down every command line and returned message for further reference.
I wonder if that’s enough stack of Swiss Cheese to prevent such an accident from happening again. Hopefully they expand on the “we should calm down” to not be limited to saying “calm down please”. Calming down is quite difficult when you’re in the throes of something, and proper procedures tend to be put aside for the sake of pressure relief.
I wish them swift recovery from their predicament.
I like that errors can be quantified as it removes some doom ambiguity from mistakes.
The first company I worked for sold access to data on a per minute basis and it was like $1.50/minute or something (dialup access in the 90s to an interactive ClarionDB session).
So any outage would be quantified by some curious programmer. I remember even small outages being quantified in hot washes as 10 minutes x 2000 users x $1.5 = $30k outage.
I remember one of my bosses was nicknamed “Commander Crash” as he once crashed the whole system for hours based on a single live config file edit.
No one was fired but it was actually liberating knowing how much mistakes cost.
The first company I worked for sold access to data on a per minute basis and it was like $1.50/minute or something (dialup access in the 90s to an interactive ClarionDB session).
So any outage would be quantified by some curious programmer. I remember even small outages being quantified in hot washes as 10 minutes x 2000 users x $1.5 = $30k outage.
I remember one of my bosses was nicknamed “Commander Crash” as he once crashed the whole system for hours based on a single live config file edit.
No one was fired but it was actually liberating knowing how much mistakes cost.
The report reads like the OptiFi folks were trying to figure out how the different bits of the puzzle on the fly. Did these folks even know how to use their tools?
Towards later in the article they say "Here it turned out that we didn’t really understand the impact and risk of this closing program command line. ‘solana program close’ is actually for closing the program permanently and sending the SOL tokens in the buffer account used by the program back to the recipient wallet."
What?!
Towards later in the article they say "Here it turned out that we didn’t really understand the impact and risk of this closing program command line. ‘solana program close’ is actually for closing the program permanently and sending the SOL tokens in the buffer account used by the program back to the recipient wallet."
What?!
Dev edits on live, makes mistake. Nothing to really see here.
I think there are some lessons that can learned from this. For example, the command line:
The program's designers should likely have chosen the verb "shutdown", instead of "close".
solana program close
The program's designers should likely have chosen the verb "shutdown", instead of "close".
While it may not accurately represent what it does, "destroy" may be a better fit.
Even "irreversible_destroy"
this is better
solana program lose-all-money
solana program lose-all-money
A simple confirmation step with a relevant warning could possibly make developers think twice before proceeding.
From personal experience - no it won't. I have made mistakes with overriding warnings - when under pressure, hyper focused and already in the mental mode of - of course I am sure I want to do it - your slow brain is just shut down so you are moving a bit like on muscle memory.
Simply add a 1 hour unskippable countdown timer to the warning. Now you have to think about it.
If you have one hour you shouldn't be editing live anyway.
From the many depressing years I've done UX work I can promise you no amount of changes they make will protect users from their own mistakes. It is far better for your own sanity to accept people make mistakes. This is what development and staging environments are for.
Accepting that people make mistakes is one thing, but reducing those mistakes is another.
If it’s a simple change, why not do it? Though, I can see your point if the change is monumental but barely reduces rate of mistakes.
If it’s a simple change, why not do it? Though, I can see your point if the change is monumental but barely reduces rate of mistakes.
But he's a crypto dev, so let's all point and laugh.
Otherwise why the edited title that explicitly points this out?
Otherwise why the edited title that explicitly points this out?
SOL, this comedy almost writes itself.
Solana ecosystem seems to have more than it’s fair share of mishaps. Still surprised anyone would prefer it over other options
Could this be a used to launder money?
As in, you have all this crypto with made up value, and we are going to “cash out” due to a mistake, and now you have whatever we stated the value is in cash?
As in, you have all this crypto with made up value, and we are going to “cash out” due to a mistake, and now you have whatever we stated the value is in cash?
So the keys to manage this specific program were kept on disk? (~/solana-keys as per the logs).
Doesn't even look like the Solana cli supports an HSM for this usecase.
Doesn't even look like the Solana cli supports an HSM for this usecase.
Might just be a demonstration. Can just use process substitution to mimic files if necessary.
> Here it turned out that we didn’t really understand the impact and risk of this closing program command line. ‘solana program close’ is actually for closing the program permanently and sending the SOL tokens in the buffer account used by the program back to the recipient wallet.
Code executions reflect intentions, transactions are final. As promised on a blockchain. Let's see if it stays that way, wouldn't be the first time if not.
[deleted]
it's not the dev's mistake, it's the whole unnecessary complicated and bullshit of this solana network and most other blockchain networks that are put up by incompetent developers.
Crypto techs don't understand crypto tech - story at 11
And nothing of value was lost.
Just another example of how useless crypto is.
rm -rf /
Whoops, I lost everything! Computers are useless.
Whoops, I lost everything! Computers are useless.
You can recover from that by restoring from backups.
In the crypto world all changes are permanent and immutable.
There is no recourse.
That’s the difference.
In the crypto world all changes are permanent and immutable.
There is no recourse.
That’s the difference.
But there is, even in cases like this. Circle could issue new USDC for example. No technical barrier to a remedy like that.
I mean that won't work, because --preserve-root is on by default. In the same way, eventually crypto may acquire the same checks and balances that other industries have, but it'll take a lot of costly mistakes to get to that point.
Preserve root exists now, but it didn’t exist in the past.
That's my point. This is the early days for crypto too. I'm not a fan of crypto, but it's clearly going to learn from these mistakes and develop the same kind of safety guarantees over time.
I mean look at online banking - in the UK it's only in the last few _years_ that we've been required to do 2fa when buying things online. The earliest online banks didn't even have 2fa to log in and move money around, and at some point HTTPS didn't even exist so everything was sent in plaintext. SMS is still mostly plaintext. Things evolve.
I mean look at online banking - in the UK it's only in the last few _years_ that we've been required to do 2fa when buying things online. The earliest online banks didn't even have 2fa to log in and move money around, and at some point HTTPS didn't even exist so everything was sent in plaintext. SMS is still mostly plaintext. Things evolve.
-> How useless SOL is
When are we going to admit that DeFi is a regression, not progress, versus the status quo financial system?
Crypto is all about disintermediation. But that doesn't work when people's money / savings are on the line.
So we can easily tell exactly where this leads: Once enough pain has been sustained through errors like this, DeFi code bugs, fraudulent transfers, etc... a whole industry of HUMANS will pop up that will (a) become an intermediary in crypto/DeFi transactions to ensure that fraud, errors, etc are prevented; (b) introduce mechanisms to revert transactions (controlled by said HUMANS); (c) perform the massive paperwork on compliance (KYC, register big transaction, check against blacklists, etc)
Is there really a different outcome?
Crypto fans claim finance is an overbloated industry and they are here to disrupt it. In reality all they are doing is recreating exactly the same system but with one more layer of indirection (instead of transferring dollars you transfer a stablecoin that points to a dollar).
Will there be exciting use cases for crypto at the edges? maybe. Will it 'disrupt' the broader financial system, in the sense that it will make it better/faster/cheaper? I don't see it.
Crypto is all about disintermediation. But that doesn't work when people's money / savings are on the line.
So we can easily tell exactly where this leads: Once enough pain has been sustained through errors like this, DeFi code bugs, fraudulent transfers, etc... a whole industry of HUMANS will pop up that will (a) become an intermediary in crypto/DeFi transactions to ensure that fraud, errors, etc are prevented; (b) introduce mechanisms to revert transactions (controlled by said HUMANS); (c) perform the massive paperwork on compliance (KYC, register big transaction, check against blacklists, etc)
Is there really a different outcome?
Crypto fans claim finance is an overbloated industry and they are here to disrupt it. In reality all they are doing is recreating exactly the same system but with one more layer of indirection (instead of transferring dollars you transfer a stablecoin that points to a dollar).
Will there be exciting use cases for crypto at the edges? maybe. Will it 'disrupt' the broader financial system, in the sense that it will make it better/faster/cheaper? I don't see it.
> a whole industry of HUMANS will pop up that will (a) become an intermediary in crypto/DeFi transactions to ensure that fraud, errors, etc are prevented; (b) introduce mechanisms to revert transactions (controlled by said HUMANS); (c) perform the massive paperwork on compliance (KIC, register big transaction, check against blacklists, etc)
When they do that they'll realize they no longer get any benefit from the underlying asset being a cryptocurrency (since it is no longer trustless as the humans are able to reverse transactions) and will switch back to a trust-based system with a good old database as the ledger.
When they do that they'll realize they no longer get any benefit from the underlying asset being a cryptocurrency (since it is no longer trustless as the humans are able to reverse transactions) and will switch back to a trust-based system with a good old database as the ledger.
Inevitably when people discuss Bitcoin someone shows up and calls it a ponzi scheme.
Defi is just the market hearing that over and over again and saying, "OK. I heard you like ponzi schemes..." They all already have humans throughout--- if nothing else, in the form of the people walking off with the windfalls.
The fact that the tech keeps blowing up is mostly just an artifact that the tech is just there as obfuscation for the fraud that underlies and motivates the enterprise. The technology is not well thought out because it doesn't need to be, it's not managed by people with high technical expertise because people with such expertise see through the schemes and can find better things to do with their time than to help rip people off. Given the economic or centrally trusted points of failure-- a competent and ethical engineer would usually tell you adding smart contract gunk to these schemes is an unacceptable source of risk without meaningful benefit (except perhaps as pretext to hide from law enforcement) and so the systems the world gets are the ones built by people who were less than competent and ethical.
Apologies to the rare few things under that banner that aren't fraud -- but they're part of a lemon market where they can't be distinguished from their more fraudulent compatriots, so that kind of guilt by association is inevitable.
Defi is just the market hearing that over and over again and saying, "OK. I heard you like ponzi schemes..." They all already have humans throughout--- if nothing else, in the form of the people walking off with the windfalls.
The fact that the tech keeps blowing up is mostly just an artifact that the tech is just there as obfuscation for the fraud that underlies and motivates the enterprise. The technology is not well thought out because it doesn't need to be, it's not managed by people with high technical expertise because people with such expertise see through the schemes and can find better things to do with their time than to help rip people off. Given the economic or centrally trusted points of failure-- a competent and ethical engineer would usually tell you adding smart contract gunk to these schemes is an unacceptable source of risk without meaningful benefit (except perhaps as pretext to hide from law enforcement) and so the systems the world gets are the ones built by people who were less than competent and ethical.
Apologies to the rare few things under that banner that aren't fraud -- but they're part of a lemon market where they can't be distinguished from their more fraudulent compatriots, so that kind of guilt by association is inevitable.
If it recreates the system but in a more efficient manner that might be good enough. Banks kind of suck anyway. it is worth a try.
Crypto is already an "industry of humans". OpenSea is a centralized company, run by humans, and they sometimes need to intermediate when there are bugs or problems in their platform that causes users to lose funds. They also have to comply with the laws of whatever country they are registered in. If an OpenSea developer fat fingered their contract deployment, it would cause massive losses.
But what does that matter? The point of decentralization is not to abolish humans, services, platforms, user-experience designs. The blockchain acts as a neutral base layer, and competing services can operate on top of the same shared data. OpenSea has limited control over the NFTs being listed on it, which is why you can buy and sell the same assets on other platforms.
But what does that matter? The point of decentralization is not to abolish humans, services, platforms, user-experience designs. The blockchain acts as a neutral base layer, and competing services can operate on top of the same shared data. OpenSea has limited control over the NFTs being listed on it, which is why you can buy and sell the same assets on other platforms.
Can you change the title of this submission to match the title of the post please? Normally people change titles when the original is sensationalised however the opposite is true in this case.
Sounds like they only lost $661k worth, which is peanuts by comparison to most other crypto hacks/losses, probably didn't destroy the entire company since they're compensating their users, and not really worthy of the sensational title.
Apparently 95% of that money was by the company itself or employees of the company. So the loss for normal users is very limited.
I can take a box containing 1001 conkers, sell one to you for $1000, then claim that I have $1m in my bucket.
What was lost was not money, but electronic conkers of which a small number people paid real money for.
What was lost was not money, but electronic conkers of which a small number people paid real money for.
The currency was USDC, which is a stablecoin pegged at $1 by Circle (www.circle.com) who are generally held to be reputable, so it very much was real money.
It sounds like a bunch of people are going to pay for the cost to users of their own mistake out of their own pocket, so if they follow through with that we should probably applaud the move?
If they welch on it that’s another matter, but the stated time is tomorrow so…so far so good?
If they welch on it that’s another matter, but the stated time is tomorrow so…so far so good?
Tweet says 95% of the $661k funds belonged to a team member. It's unfortunate in any case.
https://twitter.com/OptifiLabs/status/1564367455220219904
https://twitter.com/OptifiLabs/status/1564367455220219904
Changed now. Submitted title was "Crypto Dev Enters Wrong Command, Destroys Entire Company"
thro388(1)
good news. we need more of this.
I’m quite sure the hacker who pressed the button feels plenty bad enough already, and I hope the people around them were as kind as the people around me were.