MySQL at Uber(uber.com)
uber.com
MySQL at Uber
https://www.uber.com/blog/mysql-at-uber/
73 comments
Immediately something felt off when I saw the AI generated cover photo. Hard to describe, but I immediately profile a page online when I see clear AI slop. I wouldn't want to taint something that I had taken the time to write with something so... low effort.
More posts about mysql from Uber - it is interesting to compare state of Mysql in Uber with 2016.
* Why Uber Engineering Switched from Postgres to MySQL (2016) https://news.ycombinator.com/item?id=26283348
* Upgrading Uber's MySQL Fleet https://news.ycombinator.com/item?id=41836748
* Why Uber Engineering Switched from Postgres to MySQL (2016) https://news.ycombinator.com/item?id=26283348
* Upgrading Uber's MySQL Fleet https://news.ycombinator.com/item?id=41836748
The comments are pretty interesting. There was a discussion about how different they were trying to make the public image about their tech look compared to their actual usage - supposedly to entice more engineers to join.
I think its interesting to attach any company prominently to a database technology since theoretically there would be varied use cases across an org like uber which would likely want different technologies depending on those use cases. Of course they might just have 50 other articles like this for all the other tech they use.
I think its interesting to attach any company prominently to a database technology since theoretically there would be varied use cases across an org like uber which would likely want different technologies depending on those use cases. Of course they might just have 50 other articles like this for all the other tech they use.
Yup Netflix is similar. Your tech is not that hard and the good engineers can see through the BS.
Streaming video at Netflix's scale in the year 2010 was hard though?
I mean sure, it's a lot easier nowadays - but that's mostly down to cloud providers replacing half of the server-side challenge with a big bill and fiber making the last mile easier too.
You're massively underestimating the challenge of transferring these vast amounts of data without interrupting service for buffing etc that they had to solve back then
I mean sure, it's a lot easier nowadays - but that's mostly down to cloud providers replacing half of the server-side challenge with a big bill and fiber making the last mile easier too.
You're massively underestimating the challenge of transferring these vast amounts of data without interrupting service for buffing etc that they had to solve back then
Pornhub solved it much earlier with PHP
tbf it’s easier if you only need to serve the first 5 minutes of a video
According to Pornhub's 2024 Insights blog post, (nope, not gonna provide an URL, use a search engine :))"
"...making the average engagement time per visit 9 minutes and 40 seconds"
ah I know the blog post ;) I just thought it was ~5 minutes, but its ~10 minutes ;)
still way easier to do that than netflix ;)
Another interesting one about their Storage Platform (which includes MySQL): https://www.uber.com/blog/odin-stateful-platform
> The [Odin] platform supports 23 technologies, ranging from traditional online databases such as MySQL® and Cassandra® to advanced data platform technologies, including HDFS™, Presto™, and Kafka®.
> The [Odin] platform supports 23 technologies, ranging from traditional online databases such as MySQL® and Cassandra® to advanced data platform technologies, including HDFS™, Presto™, and Kafka®.
[deleted]
Every six months, I explore switching from MySQL to something new for a more modern tech stack. However, MyRocks (https://docs.percona.com/percona-server/8.4/myrocks-index.ht...) is truly impressive. It allows me to efficiently compress my text-rich rows.
Read the parent post to this one, AI slop has an uncanny valley associated with it. Somehow it sticks out like a sore thumb but i can't put my finger on why.
I don't know, I didn't spot that this was AI generated. Perhaps because there's some truth that MyRocks is actually really good at compression.
Back when I was exploring migrating from TokuDB to MyRocks the only problem with it was that it didn't have a file per partition, meaning if you were doing retention you couldn't just drop old daily partitions cheaply.
Back when I was exploring migrating from TokuDB to MyRocks the only problem with it was that it didn't have a file per partition, meaning if you were doing retention you couldn't just drop old daily partitions cheaply.
@bratao may not be a native English speaker.
Indeed, per their profile they're in Brazil, which is not a guarantee of that but does make it likely.
Didn't ring any alarm bells for me. I suspect you're not the target audience of this post.
I’m amazed how I’ve worked with all of the critical technologies in this article, I’ve dealt with the same or similar concerns, and yet the author or authors have written this engineering blog post in such a way as to not convey anything meaningful at all.
One part of it is the constant talk of high level abstract infrastructural pieces, and the other is bad product or concept naming.
Odin, “the controller,” the constant obsession with certain engineering orgs to use words like “plane,” and likewise, “fabric” was used at a previous org I worked for.
I’m sure Uber is doing Real™ Work, but this kind of crap sets off all my wank and bullshit alarms.
It’s just clients talking to servers talking to servers talking to proxies talking to servers talking to databases talking to replicas. Can you please stop with the false high engineering bullshit?
One part of it is the constant talk of high level abstract infrastructural pieces, and the other is bad product or concept naming.
Odin, “the controller,” the constant obsession with certain engineering orgs to use words like “plane,” and likewise, “fabric” was used at a previous org I worked for.
I’m sure Uber is doing Real™ Work, but this kind of crap sets off all my wank and bullshit alarms.
It’s just clients talking to servers talking to servers talking to proxies talking to servers talking to databases talking to replicas. Can you please stop with the false high engineering bullshit?
the plane separation is important though, you want to ensure your control messages always go through, like how servers have a dedicated line for management.
Mirror with working images: https://web.archive.org/web/20250130201835/https://www.uber....
This is ai written garbage
I ran this through ChatGPT and it found a few grammatical errors - that I agree with - and awkward wording. Usually AI generated text doesn’t have these types of errors.
It reads like a standard corporate blog post. You can find plenty of those on AWS blogs that were written before LLMs were publicly available.
It reads like a standard corporate blog post. You can find plenty of those on AWS blogs that were written before LLMs were publicly available.
Your AI detector seems to be in need of recalibration.
I think you feel that way because it's written more in a whitepaper tone instead of a blog article tone. If you're not deeply mired in distributed relational database systems, you likely won't get anything out of it.
It's not written like a "fable" as most technical blog articles that do well on HN tend to be. There is no great philosophical life-insight that the authors stumbled upon while doing this mundane technical thing.
It's not written like a "fable" as most technical blog articles that do well on HN tend to be. There is no great philosophical life-insight that the authors stumbled upon while doing this mundane technical thing.
I ran the text through several AI detectors and they all returned 0% AI 100% human.
The cover image is however:
> Cover Photo Attribution: The cover photo was generated using OpenAI ChatGPT Enterprise.
The cover image is however:
> Cover Photo Attribution: The cover photo was generated using OpenAI ChatGPT Enterprise.
[deleted]
this comment is ai written garbage
[deleted]
When people say MySQL nowadays, do they really mean MySQL, or MariaDB?
IME MySQL. Folks using MariaDB aren't shy about saying so.
In shops that have been on MySQL for thirty years, people probably continue to say "MySQL" even though it's MariaDB. They have things named after MySQL, like host names, configuration files, lines in configuration files, discussion channels and mailing lists, and whatever else.
Page is 404-ing now.
So if these clusters are using binlog replication, do they just ignore the possibility of lost writes and inconsistent data after a failover?
That all depends on the setup. The "standard" setup (not specific to MySQL) is:
- Single Write Leader per partition
- Backup Write Leader that is setup with synchronous replication (so WL -> WLB and waits for commit)
- Read Followers all connected asynchronously using either binlog replication (not recommended anymore) or GTID-based row replication (recommended)
In the above scenario, the odds of loss are pretty small since the Write Leader has a direct backup, and any of the Read Followers can be promoted to a Write Leader/Backup. DDIA calls the above semi-synchronous replication, although MySQL now supports a similar-but-slightly different version out of the box: https://dev.mysql.com/doc/refman/8.4/en/replication-semisync...
- Single Write Leader per partition
- Backup Write Leader that is setup with synchronous replication (so WL -> WLB and waits for commit)
- Read Followers all connected asynchronously using either binlog replication (not recommended anymore) or GTID-based row replication (recommended)
In the above scenario, the odds of loss are pretty small since the Write Leader has a direct backup, and any of the Read Followers can be promoted to a Write Leader/Backup. DDIA calls the above semi-synchronous replication, although MySQL now supports a similar-but-slightly different version out of the box: https://dev.mysql.com/doc/refman/8.4/en/replication-semisync...
Terminology-wise, that's not quite right. "Binlog replication" is the general term for all built-in MySQL logical replication, including all formats (statement, mixed, row); positioning via coordinates or via GTID; async or semi-sync or group replication.
Among AWS users, "binlog replication" is often contrasted against Aurora's system, which uses physical replication instead of logical.
When using binlog replication, you are correct that GTID positioning and row-based replication are strongly recommended and widely used.
Regarding semi-sync replication: that's been a MySQL feature for over a decade now, and there are some indications from Oracle that it may become deprecated in the future. (which is surprising, since many large MySQL users do leverage it to ensure writes cannot be lost. But it seems Group Replication is promoted more by Oracle.)
MySQL semi-sync doesn't necessarily involve the setup you've described as "standard" above. In my experience it's more common to see 2 replicas doing semi-sync ack'ing, not 1. And sometimes these are just binlog servers rather than full MySQL replicas; that's the setup Facebook adopted in 2015, although more recently they've moved to a home-grown Raft-based replication system.
Among AWS users, "binlog replication" is often contrasted against Aurora's system, which uses physical replication instead of logical.
When using binlog replication, you are correct that GTID positioning and row-based replication are strongly recommended and widely used.
Regarding semi-sync replication: that's been a MySQL feature for over a decade now, and there are some indications from Oracle that it may become deprecated in the future. (which is surprising, since many large MySQL users do leverage it to ensure writes cannot be lost. But it seems Group Replication is promoted more by Oracle.)
MySQL semi-sync doesn't necessarily involve the setup you've described as "standard" above. In my experience it's more common to see 2 replicas doing semi-sync ack'ing, not 1. And sometimes these are just binlog servers rather than full MySQL replicas; that's the setup Facebook adopted in 2015, although more recently they've moved to a home-grown Raft-based replication system.
Yes. This is in fact a CAP theorem tradeoff. Binlog asynchronous replication means they've elected to be AP. If you want zero risk of data loss for committed transactions (for single node failure), you'd replicate synchronously to one other replica. But then, both the primary and replica would need to be available and connected for the leader to accept write transactions, making it CP.
Correct, you would need to use something like galera clustering or synchronous replication to not lose writes.
I wonder if Uber has ever explored using Vitess. And they seems to be very slow in updating MySQL if they only came to 8.0 when latest is on 9.2 already.
Is it just me or all the diagram images are broken? It looks like the lazy-loading is not working. I tried on archive.org and archive.is and same issue: the images didn't load.
Edit: yeah those images have their src attribute in the form of “lh7-rt.googleusercontent.com/docsz/[VERY LONG SLUGS]” and a quick look at the dev console shows that Google returns “429 Too Many Requests” for all of them.
I guess they just copy-pasted the URLs from some Google Doc and tried to hotlink them here? Interestingly if you access these directly (instead of embedded into Uber's blog) then they display fine and Google doesn't complain with a 429.
Edit: yeah those images have their src attribute in the form of “lh7-rt.googleusercontent.com/docsz/[VERY LONG SLUGS]” and a quick look at the dev console shows that Google returns “429 Too Many Requests” for all of them.
I guess they just copy-pasted the URLs from some Google Doc and tried to hotlink them here? Interestingly if you access these directly (instead of embedded into Uber's blog) then they display fine and Google doesn't complain with a 429.
[deleted]
[deleted]
No mention of Schemaless? Was that retired?
Nah, they're both going on strong.
The new version is called Docstore though - https://www.uber.com/en-PE/blog/how-uber-serves-over-40-mill...
The new version is called Docstore though - https://www.uber.com/en-PE/blog/how-uber-serves-over-40-mill...
[deleted]
Uber you guys need to delete this, the quality of this post is absolutely embarrassing.
Too late I already archive.org'd it lol:
https://web.archive.org/web/20250130201835/https://www.uber....
Looks like Uber's ditched their expensive designers for AI on their corporate blog's lead images.
It reads like AI slop too. Not so sure why it is getting voted up.
People see uber tech blog and jump straight to upvoting.
I remember when that blog used to be top notch.
Today it just seems odd that anybody is still using MySQL. Postgres? Sure. SQLlite? Hell yeah! DuckDB? Of course. MySQL? Not so much.
Today it just seems odd that anybody is still using MySQL. Postgres? Sure. SQLlite? Hell yeah! DuckDB? Of course. MySQL? Not so much.
Absolutely not true in my experience. MySQL has its share of issues (all DBs do) but it is rock-solid when using the correct engine (InnoDB for most cases, RocksDB for high-throughput writes, Memory for caching). MySQL is very hard to beat for very high-volume OLTP workloads, both reads and writes. Its replication systems were years ahead of other systems (SQL Server, Postgres, SQLite doesn't have replication). DuckDB AFAIK is OLAP and they don't compete in the same space. Every DB system has "the things its good at" and MySQL really shines at very high-volume OLTP spread across partitions.
we are almost all in MySQL. we are also old and 500b
[deleted]
This is an embarrassment and just one more example of how far this enterprise has fallen.
Even as a consumer, I've stopped using Uber due to, of all things "biz 101", incompetent billing.
Even as a consumer, I've stopped using Uber due to, of all things "biz 101", incompetent billing.
[deleted]
Disgusting company: Uber takes 70% cut from rider's fare, drivers made only 30% on top of that Uber place those stolen earnings in the driver's tax report, not on them.
That sounds extremely improbable.
You simply cannot take 70% from the revenue coming from an employee-like agent, yet report that to the government as being that person's income. Not only would that be blatant fraud, but depending on the exact percentages and absolute amount, the employee might have to give all their pay to the government to covert the tax, and even owe some more after that.
In what country are they doing this?
You simply cannot take 70% from the revenue coming from an employee-like agent, yet report that to the government as being that person's income. Not only would that be blatant fraud, but depending on the exact percentages and absolute amount, the employee might have to give all their pay to the government to covert the tax, and even owe some more after that.
In what country are they doing this?
It has been years ago now, but when I did it this was partly true. Yes it showed up as income, but it also showed up as deductible expense.
Ah okay. What did deductible mean in that jurisdiction? Was it a straight write off, or a tax credit?
Where I am in Canada, tax deductions are shitty because they come off the bottom of your income, so to speak. That is to say, the eligible expenses are added up and the lowest tax bracket rate is applied to them. You then get that as a tax credit. That's shitty because you paid for those expenses with your post-tax dollars at your full tax rate, but the credit is for a low tax rate.
Whereas a write-off comes off your income pre-tax, and so the benefit effectively operates at the marginal tax rate, and can knock you into a lower bracket.
Where I am in Canada, tax deductions are shitty because they come off the bottom of your income, so to speak. That is to say, the eligible expenses are added up and the lowest tax bracket rate is applied to them. You then get that as a tax credit. That's shitty because you paid for those expenses with your post-tax dollars at your full tax rate, but the credit is for a low tax rate.
Whereas a write-off comes off your income pre-tax, and so the benefit effectively operates at the marginal tax rate, and can knock you into a lower bracket.
USA, in the USA deductions come off the top like your writeoffs.
Even the authors listed at end looks like AI generated.
Uber is not a company worth following for technology. It’s like a bunch of juniors using tech wrong and changing tech as a solution to a non existent problem.
I always got the impression their tech stack was designed to be as complicated as possible so they could hire juniors who thought it looked cool.
They used to produce some neat stuff. Jaeger and Cadence were quite cool.