Update on the Atlassian outage affecting some customers(atlassian.com)
atlassian.com
Update on the Atlassian outage affecting some customers
https://www.atlassian.com/engineering/april-2022-outage-update
91 comments
Bitbucket was also an acquisition (Most of their offerings were acquisitions).
https://techcrunch.com/2010/09/29/atlassian-buys-mercurial-p...
https://techcrunch.com/2010/09/29/atlassian-buys-mercurial-p...
I think Confluence is a genuinely good product. Its nice to see that even Atlassian struggle with day to day sysadmin of Atlassian tools
I read somewhere recently that confluence is write-only because search is so bad nobody will ever find what your write.
I haven’t used it in years, but that seems accurate. I could never find anything unless I knew exactly how it was worded in what I’m looking for.
I haven’t used it in years, but that seems accurate. I could never find anything unless I knew exactly how it was worded in what I’m looking for.
Putting markdown text into SQLite with FTS5 works amazingly well for search. How come they couldn't just bolt that in?
Yes, it's actually so bad that you wonder what part of the picture we're missing.
A lot of people look down on Jira. My experience was that Confluence was much worse. It turns writing a couple technical notes into a publishing journey with missteps and failures along the way.
For me Confluence dramatically improved in experience when I learned that it's possible to write files locally using $FAVORITE_EDITOR in confluencewiki format (or markdown if you don't care about macros) and post them through API.
Can you share anything more about your technique? As far as I know, Confluence hasn't supported a markup format natively for several years. I've found some tricks to allow me to author stuff in Markdown but it seems to come with a lot of limitations which ultimately mean that I have to start using the web editor as soon as I want to start collaborating with others.
In Confluence Editor, under the Macro "Plus" button, you should find an entry called "Markup" pretty far up the list, which accepts "Confluence Wiki" Markup (which is just Markdown + Macros). This works for both on-premise as well as cloud instances.
I wrote a tool that automates the posting part[1]. Unfortunately I could not solve the collaboration problem either. Setting the body of the page overwrites it.
I did see some examples of editing HTML through similar approach but never tried it myself.
[1]: https://github.com/VTimofeenko/confluence_poster
I did see some examples of editing HTML through similar approach but never tried it myself.
[1]: https://github.com/VTimofeenko/confluence_poster
When they first acquired confluence it was great. Now I can’t stand it. Is so slow now. So clunky.
I had Jira inflicted on me before Atlassian was big and Jira was terrible back then too. I always liked FogBugs.
>I always liked FogBugs
You lost me there. In no way was FogBugz a better product. It was at best mediocre by 2010.
You lost me there. In no way was FogBugz a better product. It was at best mediocre by 2010.
Current user of FogBugz, and prior user of JIRA.
Jira is much better. We also use SharePoint and Confluence is so much better.
Honestly I get the criticism of both JIRA and Confluence but till you've used the other big names in that space you don't realise how bad it can get.
Jira is much better. We also use SharePoint and Confluence is so much better.
Honestly I get the criticism of both JIRA and Confluence but till you've used the other big names in that space you don't realise how bad it can get.
>Current user of FogBugz
I'm so sorry. Fogbugz has been in maintenance mode for years now, stuck in the middle of a half-finished redesign.
I'm so sorry. Fogbugz has been in maintenance mode for years now, stuck in the middle of a half-finished redesign.
Fogbugz was on the downhill as early as 2010. I always assumed that when Fogcreek started working on Trello they pulled the A team off Fogbugz.
Having dealt with the pre and post sales support quite extensively over the years, I always got the impression they were a big fish in way over it's head, and with a tail that didn't work so good.
>Jira itself has faded and been caught up with too.
Tell this to all the big corps still running a slow, bureaucratic JIRA system demanding one to update multiple tickets multiple times.
Tell this to all the big corps still running a slow, bureaucratic JIRA system demanding one to update multiple tickets multiple times.
What caught up with JIRA
I have yet to find something as good, even with all of JIRAs pain points
I have yet to find something as good, even with all of JIRAs pain points
We've been using https://linear.app/
Asana. It has its own idiosyncrasies, but it's faster, cleaner, supports things out of the box that I just gave up on in JIRA...
I've tried it before, it's just as bloated as Jira and performance suffers as a reason. If I'm going to suffer slowness anyway I might as well stick with Jira.
Having used Asana in several organisations, I think it being a suitable replacement for JIRA says more about how poor a choice JIRA was for situation than anything about Asana.
Ive tried Asana breifly a few years agoand it didn't seem to have enough to make it worth the effort to swap.
Maybe I should check it out again
Maybe I should check it out again
I don't recommend anyone to swap from JIRA. Stability is probably more important for a PM tool, but for new projects trying out something new makes sense.
Trello? OpsGenie? Statuspage?
I don't think you could attribute Trello to Atlassian. Maybe all the junk they've added to it.
The other two I've honestly never heard of but that says nothing about them as products except they don't get talked about.
Funny edit, Atlassian actually made a plugin for Jira called greenhopper that was very much like trello. But they discontinued it only to realise a few years later that they liked trello and paid a massive amount of money for it. If that doesn't exemplify incompetence of a companies strategy / directorship, then I don't know what does.
The other two I've honestly never heard of but that says nothing about them as products except they don't get talked about.
Funny edit, Atlassian actually made a plugin for Jira called greenhopper that was very much like trello. But they discontinued it only to realise a few years later that they liked trello and paid a massive amount of money for it. If that doesn't exemplify incompetence of a companies strategy / directorship, then I don't know what does.
Maybe it is my enterprise hat, but they always seem to miss something.
The report is deliberately obscure.
My interpretation is... Atlassian Cloud is hosted in shards where a shard is allocated a single AWS RDS database with (say) 100 customers.
Their script was run improperly and dropped tables from 400 customers across 400 different shards (across different databases)
Their backup strategy was just to restore RDS backups (recover whole database with many customers data together)
But they can't simply restore from RDS backups due to the co-mingling of customers. As otherwise unaffected customers would now lose data.
I suspect they're manually restoring RDS backups to new DB instances and having to scrape out the data for a single customer - a process they'd not built any tooling for as it was a fault case they thought they didn't need. Possibly as running the Jira/confluence backups are very heavy and create a lot of load/disk that they can't easily accomodation in a multi tenant hosting architecture.
My interpretation is... Atlassian Cloud is hosted in shards where a shard is allocated a single AWS RDS database with (say) 100 customers.
Their script was run improperly and dropped tables from 400 customers across 400 different shards (across different databases)
Their backup strategy was just to restore RDS backups (recover whole database with many customers data together)
But they can't simply restore from RDS backups due to the co-mingling of customers. As otherwise unaffected customers would now lose data.
I suspect they're manually restoring RDS backups to new DB instances and having to scrape out the data for a single customer - a process they'd not built any tooling for as it was a fault case they thought they didn't need. Possibly as running the Jira/confluence backups are very heavy and create a lot of load/disk that they can't easily accomodation in a multi tenant hosting architecture.
If their backup strategy is as efficient as the UX of their tools, they should find what they are looking for in 2 weeks… oh wait…
At least right now they are focussed on this issue and that should stop from outputting yet more unwanted bloat in my daily ops for a while.
Guess its time to pay Asana another visit to see where they are at. Didn’t feel it was worth the move 2 years ago, maybe things have changed.
At least right now they are focussed on this issue and that should stop from outputting yet more unwanted bloat in my daily ops for a while.
Guess its time to pay Asana another visit to see where they are at. Didn’t feel it was worth the move 2 years ago, maybe things have changed.
Interesting hypothesis, and if true, I have the utmost empathy for the engineering teams that have to figure this out a way to kludge all the right data together. The thought of extracting the lost data from the sharded backups, and then merging it into a possibly a shard with good, newer customer data makes my skin crawl.
> Faulty script. Second, the script we used provided both the "mark for deletion" capability used in normal day-to-day operations (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons. The script was executed with the wrong execution mode and the wrong list of IDs.
Bad script! Bad! So faulty, running itself with the wrong list of IDs and running itself in the wrong mode.
Bad script! Bad! So faulty, running itself with the wrong list of IDs and running itself in the wrong mode.
It was faulty in the sense of allowing it to be executed with incorrect human inputs, and in such a manner that there was no recovery process in place for what happened.
Lesson learned is not have any semi-automated script dependent on human input/communication/decisions without also having a recovery plan for the worst-case scenario of said script being run with the wrong input/communication/decisions.
I doubt anyone can claim that for all such potentially dangerous scripts. But something that should be reconsidered for anything like a compliance delete script, even if the design intent is to specifically bypass the normal soft-delete process for something where it has been specifically decided that it should NOT be restoreable.
Fix in this case would be to make it a two-stage process: first an easily revertible soft-delete, evaluate the impact, and then a hard delete designed to only delete data that has already been soft-deleted. Never hard-delete active, live data directly.
Lesson learned is not have any semi-automated script dependent on human input/communication/decisions without also having a recovery plan for the worst-case scenario of said script being run with the wrong input/communication/decisions.
I doubt anyone can claim that for all such potentially dangerous scripts. But something that should be reconsidered for anything like a compliance delete script, even if the design intent is to specifically bypass the normal soft-delete process for something where it has been specifically decided that it should NOT be restoreable.
Fix in this case would be to make it a two-stage process: first an easily revertible soft-delete, evaluate the impact, and then a hard delete designed to only delete data that has already been soft-deleted. Never hard-delete active, live data directly.
Yes. Having the safe thing and very dangerous thing just be a boolean flag apart is bad. It increases the odds of the wrong flag being used at some point.
It's also not clear from the post if someone had used the "permanently delete" flag accidentally before. It's very possible they had, but hadn't ever run it on a wrong id before.
While yes, someone ultimately did hit "enter" on a command with the wrong ids and the wrong mode, the existence of such a script enabled that mistake to have enormous consequences. Calling such a script "faulty" feels reasonable. In the same way having tooling which allows you to delete prod database disks without any additional safeguards would be called faulty.
It's also not clear from the post if someone had used the "permanently delete" flag accidentally before. It's very possible they had, but hadn't ever run it on a wrong id before.
While yes, someone ultimately did hit "enter" on a command with the wrong ids and the wrong mode, the existence of such a script enabled that mistake to have enormous consequences. Calling such a script "faulty" feels reasonable. In the same way having tooling which allows you to delete prod database disks without any additional safeguards would be called faulty.
It helps to write post mortems like this. If you start writing them in a way that suggests blame, you miss out on process issues and everyone clams up the next time it's post mortem time.
“A script was executed incorrectly” does not blame anyone. Saying a script was faulty obfuscates what happened.
It is faulty though? There should be more that's required to accidentally delete data than a manually specified switch.
I mean, if it’s got flags to represent what you’re asking it to do and you set the wrong flag that’s not a script issue. I can run rm -rf / and that’s not a fault of rm, that’s a behind the keyboard issue.
On most modern systems you can't, not without going out of your way to specify --no-preserve-root.
It sounds like they have a "type system" problem, e.g. you shouldn't be able to give a script an ID for a "cloud site" when it was expecting an ID for an application. That is, this sort of outage should not have occurred because it should not have been representable to begin with.
It sounds like they have a "type system" problem, e.g. you shouldn't be able to give a script an ID for a "cloud site" when it was expecting an ID for an application. That is, this sort of outage should not have occurred because it should not have been representable to begin with.
It's a process issue that a script like that is ~commonly used on production data. Processes should be in place where that's essentially impossible to do wrong. "oops I passed the wrong flag" isn't user error, that's system error that it was set up like that.
Right—and that's not a "faulty script", that's a "faulty process for running scripts".
No. That is a fault of rm that it makes it so easy to do something that in 9.99999% of the cases isn’t what the user wants to do.
In fact, recognizing that is precisely why you can’t even do that easily anymore. Setting aside that you probably need to sudo the function, rm prevents you from running the command unless you pass an additional long flag that basically say you really really want to do this.
Programs which expect perfect user input, and allow minor user mistakes to lead to major negative outcomes are faulty programs.
In fact, recognizing that is precisely why you can’t even do that easily anymore. Setting aside that you probably need to sudo the function, rm prevents you from running the command unless you pass an additional long flag that basically say you really really want to do this.
Programs which expect perfect user input, and allow minor user mistakes to lead to major negative outcomes are faulty programs.
Adding a hardcoded extra check for / to rm feels like theater because it barely makes it more safe. It's nearly as damaging to most Linux systems to remove /bin or /usr or any number of other directories. If you wanted to actually solve this problem, it seems like you would need to add some kind of "dangerous-to-remove-this" metadata in the filesystem.
Nonetheless, even with rm you can override the check you mentioned with a flag. In principle, that seems to support the post you are referring to, not refute it (not in detail but in spirit).
Nonetheless, even with rm you can override the check you mentioned with a flag. In principle, that seems to support the post you are referring to, not refute it (not in detail but in spirit).
[deleted]
rm literally has a specific handling of / that requires the `--no-preserve-root` flag to allow you to run that. Software should generally warn before being super destructive, and it's usually best if the process for doing something super destructive and day-to-day use are very different.
Also this - that the script runs in one step!
When I do big deletions in an automated fashion, I always make it a two-step process.
The first step identifies exactly the items which will be deleted.
This gives you the user a chance to look at it and say, "Why are we deleting 100x as many entries as we expected?"
In many tools, this is as simple as running it twice, the first time with `--dry-run`.
When I do big deletions in an automated fashion, I always make it a two-step process.
The first step identifies exactly the items which will be deleted.
This gives you the user a chance to look at it and say, "Why are we deleting 100x as many entries as we expected?"
In many tools, this is as simple as running it twice, the first time with `--dry-run`.
I'd guess they mean in in the "blameless post mortum" way, in that it wasn't some Random User's fault, it was a total process breakdown that led to said hands-on-keyboard executing with the wrong information and safeguards in place
Okay, so the script was faulty and the data it was running on was faulty. But I see no mention of testing. In my entire I’ve never seen initiatives like this not have a staging environment with dry runs and validation. This is bad engineering practice.
There’s some assumption here that the script directly updated/deleted customer data.
In a microservices-centric architecture, it’s entirely possible that the script was used to orchestrate the change across a number of other systems, and an update on one of them led to to undefined behaviour and/or unforeseen consequences.
In a microservices-centric architecture, it’s entirely possible that the script was used to orchestrate the change across a number of other systems, and an update on one of them led to to undefined behaviour and/or unforeseen consequences.
This is what happens when DRY gets taken too far, deduplicating similarly named or conceptualized operations with vastly different use-cases.
Good on them for doing a relatively detailed, honest and public review like this. But goddamn that recovery time is absolutely unacceptable for a modern mission-critical product like this, especially since they've essentially forced everyone onto their cloud platform. Not having the ability to automatically restore backups in 2022, when you're a MAJOR SaaS provider that stores massive amounts of essential data is just mind boggling to me.
Hopefully this event has enough of an impact for them internally (and financially) that they seriously reevaluate things and maybe invest in some SRE capability, and better tooling.
Hopefully this event has enough of an impact for them internally (and financially) that they seriously reevaluate things and maybe invest in some SRE capability, and better tooling.
I think what they're implying is that they can restore _everyone's_ data quickly (i.e. 'usual' infra failure), and one Jira instance's data fairly quickly (but some manual steps), but the problem is they nuked a large subset of customers while everything else kept running. This means that they can't do a complete automatic restore, and instead have to deal with it customer at a time.
I'm sure that they're going to make their customer at a time thing properly automatic now... Still a big failure, but a little more subtle than you make out.
(disclaimer - worked for Atlassian years ago)
I'm sure that they're going to make their customer at a time thing properly automatic now... Still a big failure, but a little more subtle than you make out.
(disclaimer - worked for Atlassian years ago)
The thing is that the data was wiped as part of a "Compliance Delete", albeit a wrongly executed one.
I am actually more surprised that they are able restore the data at all. If you can recover from a Compliance Delete, is it really a Compliance Delete?
I am actually more surprised that they are able restore the data at all. If you can recover from a Compliance Delete, is it really a Compliance Delete?
If immutable backups can be nuked by a single compliance delete request why bother making backups at all? All it takes is a delete request per day to get rid of them.
By removing data and only keeping a 30 day backup any compliance deletes still simply take ~a month to complete.
By removing data and only keeping a 30 day backup any compliance deletes still simply take ~a month to complete.
I think most teams delete the live data from the database then wait for the backups to expire after N days.
It's clearly not mission critical, Atlassian has just demonstrated that.
"Move fast and break things" and "mission critical" are opposing forces that just do not mesh.
My team and I run emergency services communications infrastructure with 99.999% availability requirements and our company has to pay an abatement if we don't maintain those levels. That's actually mission critical. We don't run ANY script or SQL command in production that deletes data without having tested it in our dev/test environment, documented the steps of the change and the results of testing, having the change reviewed internally, and approved by the government.
We can get these changes approved and into production in around a week normally. "Move fast and break things" is great, until the wrong thing breaks.
My team and I run emergency services communications infrastructure with 99.999% availability requirements and our company has to pay an abatement if we don't maintain those levels. That's actually mission critical. We don't run ANY script or SQL command in production that deletes data without having tested it in our dev/test environment, documented the steps of the change and the results of testing, having the change reviewed internally, and approved by the government.
We can get these changes approved and into production in around a week normally. "Move fast and break things" is great, until the wrong thing breaks.
The list of affected data that was wiped out by the compliance delete and now requires recovery also includes the metadata in their orchestration system, and what sounds like other multi-tenant data stores. It's not just restoring a single, entire PostgreSQL database from backups.
Lesson learned is not not have any semi-automated script dependent on human input/communication/decisions without also having a recovery plan for the worst-case scenario of said script being run with the wrong inputs.
Lesson learned is not not have any semi-automated script dependent on human input/communication/decisions without also having a recovery plan for the worst-case scenario of said script being run with the wrong inputs.
I agree that the timeline is not really acceptable. I am not sure anything mission critical should be hinged off a ticketing system, though. At best, things should be reported to the ticketing system, but never triggered from it as some sort of source of truth.
> Instead of providing the IDs of the intended app being marked for deactivation, the team provided the IDs of the entire cloud site where the apps were to be deactivated.
This reminds me of the time I was tasked with changing the gender of a user to female on the production database (since the feature was not available through an interface at the time) and so I opened SQL Explorer on our live database and ran a quick update. Except I forgot the UserID param and accidentally changed over a million registered users to female.
A colleague came up with the idea to use the person's title as a hint to repopulate the database, which worked for Mr, Mrs and Ms.
I apologise now to all the doctors that found they had undergone nonconsensual gender reassignment.
This reminds me of the time I was tasked with changing the gender of a user to female on the production database (since the feature was not available through an interface at the time) and so I opened SQL Explorer on our live database and ran a quick update. Except I forgot the UserID param and accidentally changed over a million registered users to female.
A colleague came up with the idea to use the person's title as a hint to repopulate the database, which worked for Mr, Mrs and Ms.
I apologise now to all the doctors that found they had undergone nonconsensual gender reassignment.
All databases should offer an --i-am-a-dummy mode, and probably enable it by default in clients. It prevents you from running update or delete statements that don't have a where clause.
https://dev.mysql.com/doc/refman/8.0/en/mysql-tips.html#safe...
https://dev.mysql.com/doc/refman/8.0/en/mysql-tips.html#safe...
That's why some databases have a "where condition required on update" so you won't hit that wall (assuming there were no other where conditions involved).
If one record needs to be changed, I typically, just manually do it via the UI :)
It's an important highlight that Atlassian is deprecating all of their on-premise offerings and forcing everyone to the cloud because it's "the future". And then promptly demonstrated how poor of stewards of the cloud they are.
Not true at all. DC is still available for the next decade. It's just not at a price that smaller sized customers can afford to pay, so your point does apply to that group.
They tried to sell my company on their cloud offering but once we described our current deployments scale they backed away since it was larger then they had the capability to support.
noicebrewery(2)
https://www.atlassian.com/engineering/april-2022-outage-upda...
"Something went wrong. We're moving mountains to get it sorted."
Works as axpected.
"Something went wrong. We're moving mountains to get it sorted."
Works as axpected.
I just got the same! More than a bit worrying.
Edit: After reviewing the snippets below looks like the script has gained sentience.
Edit: After reviewing the snippets below looks like the script has gained sentience.
> Additionally, the majority of restored customers have had no data loss, while some have reported data loss for up to 5 minutes prior to the incident.
This phrasing is ambiguous on a major question.
If the reality is that no customers had more than 5 minutes' data loss from prior to the incident, then it'd be good to say that.
This phrasing is ambiguous on a major question.
If the reality is that no customers had more than 5 minutes' data loss from prior to the incident, then it'd be good to say that.
I'll just assume some customers lost a lot more :)
Not to mention all the data that was lost, because you can't create it, because the servers are down.
Not to mention all the data that was lost, because you can't create it, because the servers are down.
Question for Atlassian Customers on this site.
Does this event make you re-consider using their software?
Also, I'm not clear if the site was down for all customers for a few days and now only 400 (0.18%) have an issue or if it was only down for only 400 customers. After looking at many tweets, comments here and reddit it looks like it was down for everyone but I'm not sure.
Does this event make you re-consider using their software?
Also, I'm not clear if the site was down for all customers for a few days and now only 400 (0.18%) have an issue or if it was only down for only 400 customers. After looking at many tweets, comments here and reddit it looks like it was down for everyone but I'm not sure.
It was down only for the 400. But some of them were bigger than others. In general though, it's natural for the affected customers to he more vocal than the unaffected customers, and given a large portion of the target market use Reddit/Hacker News, it's to be expected that you'll see a lot of comments.
Thanks!
Interesting, looks like my original estimate that they were trying to restore each customer individually a backup wasn't even that far off.
Still odd that they didn't have any tools in place to help them with it. Only having a way to restore an entire backup is only useful for disaster recovery, it's nigh useless to correct mistakes.
Still odd that they didn't have any tools in place to help them with it. Only having a way to restore an entire backup is only useful for disaster recovery, it's nigh useless to correct mistakes.
This ID thing makes no sense. Did the team doing the deletion take these cloud site IDs and plug then into a script expecting app IDs? And it still found and deleted sites? Or is the script like a delete anything script? Give it an ID and it will go find it in any table and delete it? That seems to be what they are implying.
Wait I'm supposed to see a post? All I'm seeing is:
Something went wrong. We're moving mountains to get it sorted. View our status page and subscribe for service updates.
Something went wrong. We're moving mountains to get it sorted. View our status page and subscribe for service updates.
What will be the best way to backup and restore cloud instances of Jira and Confluence?
Self hosted solution ftw...
Sadly not an option - Atlassian have stopped selling on-prem licences.
Not true. They only stopped selling the server edition (single node). Data center (single node or multi node) edition still exists.
Assuming you can afford the massive price hike from the current on prem pricing.
“End-to-end, it takes between 4 and 5 elapsed days to hand a site back to a customer.” Cool marketing strategy. Maybe you’ll get your data in a business week.
Their original and early Jira product was great and put them out in front, but I couldn’t name any other product of theirs except Bitbucket which is very much a second place competitor in their market. Jira itself has faded and been caught up with too.
Reading through the post and seeing the level of support some people on Twitter have been tweeting about, I would take this as a great time to look at other offerings out there.