Scaling Bitbucket’s Database(bitbucket.org)
bitbucket.org
Scaling Bitbucket’s Database
https://bitbucket.org/blog/scaling-bitbuckets-database
58 comments
Bitbucket is a perfect case for sharding. You could have one DB of users, and N DBs of user data, with the first referencing the latter, and then distribute user data among the latter. You could do this straight up with PostgreSQL without any special server-side software. You could also have a PG server using FDW to act as a proxy for all the DBs a client needs to be talking to. There are many other options too.
Does storing the LSNs in Redis and Elasticache create the same race condition they are trying to avoid or am I missing something?
Is it just me or does looking at write logs to determine read consistency just seem like a really bad way to do this vs a different load balancing strategy?
Definitely feels like bandaid, over a more robust solution like sharding their main database by repository. Or splitting off some of their tables to another database. Their load of "millions of requests per hour", doesn't sound particularly high either.
At some point their primary is going to get overloaded again by the writes. And they'll have added all of this machinery for nothing. They've also made the replicas an essential part of their stack. Whereas their previous stack could probably tolerate a certain amount of replicas going down or replica lag. They now have a system that will grow and be dependent on all the replicas being available. Until a heavy write load or heavy table alter, causes the replicas to become lagged at which point a higher percentage of traffic will go to the master and potentially cause downtime.
At some point their primary is going to get overloaded again by the writes. And they'll have added all of this machinery for nothing. They've also made the replicas an essential part of their stack. Whereas their previous stack could probably tolerate a certain amount of replicas going down or replica lag. They now have a system that will grow and be dependent on all the replicas being available. Until a heavy write load or heavy table alter, causes the replicas to become lagged at which point a higher percentage of traffic will go to the master and potentially cause downtime.
Yikes. Excellent point. Is sharding the only option here? It seems like having immensely shared Postgres is one way of solving the problem, but is that the only option?
Wondering if eg cockroachDB would be a good fit
Wondering if eg cockroachDB would be a good fit
This is a very common solution. The biggest advantage is that it allows you to use read replicas which are there for HA and otherwise idle.
It's amazing that Atlassian still exists.
Oracle offers DML redirection for active dataguard setup where a write operation on the replica database is redirected to the primary database to allow applications that make infrequent writes to actively run on the Active Data Guard replica database. Also the write operation completes when the replica has seen the write from the primary thereby eliminating the race condition avoiding such complex reengineering
A lot of databases have a concept of consistent reads. Even DynamoDB will allow you to specify consistent reads explicitly on a query and writing through DAX will take care of the scenario as you described as the read cache is immediately updated. This a common scenario.
[deleted]
Step 1: Remove Mercurial support.
Still salty about them dropping mercurial support without providing any migration tools, and planning to just delete data without any archiving.
Also salty!
For hg->git migration, https://github.com/frej/fast-export worked smoothly for me.
For hg->git migration, https://github.com/frej/fast-export worked smoothly for me.
mkdir new-git-repo
cd new-git-repo
git init
fast-export/hg-fast-export.sh -r ../path-to-old-hg-repo
(probably want to translate .hgignore to .gitignore and commit it)
git remote add origin git@path_to:git_server
git push -u origin master
git push --allYes, it is fairly easy to convert the repo, but
1) this afaik couldn't be done inside the BB project, so one would have had to create a new repo and lose bug history etc.
2) It's a trust thing. If BB kills repos once, who doesn't say they won't kill others tomorrow?
3) Comfort: Import to GitHub is a click away
1) this afaik couldn't be done inside the BB project, so one would have had to create a new repo and lose bug history etc.
2) It's a trust thing. If BB kills repos once, who doesn't say they won't kill others tomorrow?
3) Comfort: Import to GitHub is a click away
Same here. I don't love that they aren't going to support Mercurial anymore. But deleting all our existing projects without providing a path for us to migrate or archive? I can't trust them anymore.
I'm also salty. What's your plan? I'm either gonna migrate to git (and use a different host), or self-host something like hetapod.net
I went to sourcehut. I am very happy, but during the process I ended up actually having to learn a bit of git. Future projects will probably be git.
Yeah I'm moving to git and github. I always pteferred mercurial for what I maintain are good reasons not just of personal taste, but I've been swimming against the tide for too long and it makes sense to go with git and github.
If github goes downhill, at least I'll be in good company next time!
If github goes downhill, at least I'll be in good company next time!
GitHub’s import feature worked well for me. You’re obviously going to lose the wiki, issues etc. But at least you’ll keep your repo history, branches, ...
"Bitbucket uses PostgreSQL with one primary read-write database and N read-only replicas."
That's really neat! I wonder, could you store the PostgreSQL log sequence numbers in a cookie to save the round-trip to Redis?
I thought this too, but that then creates a likely unreasonable burden on the client. Not all user agents expect cookies either, think of all the varieties REST clients. Sounds like they would also need to internally reference the lsn too, so being able to look it up internally would be valuable.
Feels like there is something there tho in terms of approach, esp since they are keying it on user id.
Feels like there is something there tho in terms of approach, esp since they are keying it on user id.
Yea I'm not taking advice from a service that's down once a week
Did you actually attempt to read the article or did you just come here to spew out needless negativity?
Have you ever actually used Bitbucket in production? It has absurd latencies for every operation from website to even a git push taking full seconds to respond.
It’s hard to take them seriously when that’s the tableau upon which they are advising us to build systems.
It’s hard to take them seriously when that’s the tableau upon which they are advising us to build systems.
This.
A number of years ago I had all of my repos on bitbucket, but recently I’ve pulled everything as their peering can impact large git pulls and were unreliable. The transition with the account changeover was enough to make me jump ship to github completely.
I don't think it's needless. I think the fact that Bitbucket is profoundly second-rate in is quite relevant when discussing their fluffy blog posts about engineering.
I’m not a big fan of Atlassian in general but the blog post provides a reasonable solution to an engineering problem. It doesn’t seem like a fluff piece.
Bitbucket (mostly via Atalassian) worked very hard to chase everyone off their platform in the vain hopes that the OSS community would start using Jira. Clearly they bet wrong, but unfortunately they aren't going to get any users back for a very long time.
I've already moved 100 repos from BB over to Github. Free private repos on Github were not enough to get me to go to the trouble of moving, but then BB deprecated Mercurial, so there's no longer any reason to stay. I'm gone.
I just went through this as well. And I discovered that GitHub can import private bitbucket mercurial projects. That made it super easy.
Agreed. Turned out to be much easier than I expected.
I also don't see anything groundbreaking about Bitbucket as a Git repo interface. GitHub is miles ahead in both polish and mindshare.
Free private repos used to be the one thing keeping me (and from conversations, several people I know) on Bitbucket and once GitHub offered those as well, there was no incentive to stay. The Mercurial support was probably another unique feature they had from way back but they seem to have decided to remove that too.
Free private repos used to be the one thing keeping me (and from conversations, several people I know) on Bitbucket and once GitHub offered those as well, there was no incentive to stay. The Mercurial support was probably another unique feature they had from way back but they seem to have decided to remove that too.
yes, at the beginning, we thought github and bitbucket would be in a healthy competition for features and polish. But then Atlassian bought Bitbucket, and their agenda was, "make BB as awful as possible so we can steal whoever into our crappy corporate software" and that was that. I had to move not just because the issue tracker was garbage but also because Bitbucket's UX became horrible, not just the slowness but also the layout was very broken, having to read diffs inside of a tiny scrolling pane and things like that.
Yep, it was private repos for me. Once Gitlab came about and offered free private repos I bailed as quickly as I could from Bitbucket.
Free private repos is a real upside.
Central login where you can never understand what login you are using and what information they are collecting, nor on which website you are not whether you’ve suddenly integrated Confluence into a Bitbucket instance, is a real downside. God I hate SSO’ed companies, they went all the way of Google into Youtube.
Central login where you can never understand what login you are using and what information they are collecting, nor on which website you are not whether you’ve suddenly integrated Confluence into a Bitbucket instance, is a real downside. God I hate SSO’ed companies, they went all the way of Google into Youtube.
> Free private repos is a real upside.
which isn't true anymore, as other pointed out.
Also offering Mercurial was a reason for choosing Bitbucket, but they are ending its support.
which isn't true anymore, as other pointed out.
Also offering Mercurial was a reason for choosing Bitbucket, but they are ending its support.
[deleted]
Do you have a link or two for that?
nope just some email exchanges with employees which started with begging them for the most rudimental Bitbucket issue tracker features and fixes and ended with semi-explicit admissions that their hands were tied due to the "just move to Jira" corporate directive.
See "Want a more powerful issue tracker?" at the bottom of https://confluence.atlassian.com/bitbucket/issue-trackers-22... for the documentation of this policy.
See "Want a more powerful issue tracker?" at the bottom of https://confluence.atlassian.com/bitbucket/issue-trackers-22... for the documentation of this policy.
Curious that they use LSN tracking vs making the replication synchronous. I would be curious to see the performance numbers and the reasoning behind the choice.
Synchronous replication really slows down write performance and adds in the complexity of needing to determine if a replica is down. I think very few people use it in practice.
+1, sync replication also reduces availability: any replica downtime becomes write unavailability until you decide that replica is "dead" and stop replicating to it.
There's also hybrid setups of sync+async for durability reasons, sync to a single replica so that if your primary db suddenly catches fire you have a replica that's as up to date as possible, then read replicas using async replication.
There's also hybrid setups of sync+async for durability reasons, sync to a single replica so that if your primary db suddenly catches fire you have a replica that's as up to date as possible, then read replicas using async replication.
10ms to do a redis get? What am I missing here...
And presumably query the replicas until an up-to-date one is found (based off the LSN of the user).
I’d have expected the LSN of the replicas to be in redis as well, but fair point. However even as a worst case given their number of replicas, I’d expect to be able to hit redis AND each replica in under 10ms
I’m not sure if I’ve missed something with their LSN work or if this indicates that the GET semantics horse has already escaped their proverbial barn. Within the call-response of a single request, which they seem to be talking about, none of this should be necessary. Right?
However a cascade of narrowly-spaced follow-up requests could easily catch you in this trick. Lie. Whatever you wish to call it.