TiDB – cloud-native, distributed SQL database written in Go(github.com)
github.com
TiDB – cloud-native, distributed SQL database written in Go
https://github.com/pingcap/tidb
66 comments
VC backed database company with a CLA on their "open source" project. Red flag.
Is there a community fork yet?
Thank you for the warning.
Whats the source on the VC part?
I don't share OP's hostility to VC-funded databases, but PingCAP is definitely VC-funded: https://www.pingcap.com/press-release/pingcap-the-company-be...
The VC part isn't 100% negative but it generally comes with the strings attached that highly incentivizes fast growth. Fast growth kills any company culture you have and converts you into yet another company that views all employees as cost-centers. IE. it destroys companies as good places to work. If you could get VCs to fund companies to grow slowly into stable, small to mid-sized companies I would have no problem with them.
A side aspect of this is that they also destroy any motivation for developing free software other than as a marketing tool, a way to drive growth. While that is perfectly fine for companies to do this, it has long term consequences that lead to bad behavior that mean you should be extra wary of relying on them.
A side aspect of this is that they also destroy any motivation for developing free software other than as a marketing tool, a way to drive growth. While that is perfectly fine for companies to do this, it has long term consequences that lead to bad behavior that mean you should be extra wary of relying on them.
How would you fund the open source development of a database?
TiDB is Apache licensed, that should be enough no? Because if we ban all projects from VC-backed startups, you're going to clear your tech stack pretty quickly.
TiDB is Apache licensed, that should be enough no? Because if we ban all projects from VC-backed startups, you're going to clear your tech stack pretty quickly.
I think the concern is that the company would relicense future versions, as allowed by the CLA, under pressure from VCs.
Open software licenses are generally non-revokable (unless you violate them). They can only change the license of future versions.
I'm afraid that they can always do that with or without a CLA. Apache is quite a permissive license.
I'm afraid that they can always do that with or without a CLA. Apache is quite a permissive license.
I'm not an expert in licenses, but my understanding is that if someone makes a contribution under one license, then any program including that code needs to obey the terms of that license. The CLA allows them to choose any license (well, depending on the specifics of the CLA) they would like to distribute that code under in the future (although they'd likely need to continue to make available past source to meet the terms of previously released versions.
So if I make a contribution under some very restrictive license, but sign a CLA allowing relicensing, all bets for the future are off.
So if I make a contribution under some very restrictive license, but sign a CLA allowing relicensing, all bets for the future are off.
Apache license doesn't forbid anyone to integrate your code and release the combination under proprietary licenses. The only requirements are to keep the license file and the copyright notice intact, and to state changes if you choose to publish the modified source code.
Therefore, it doesn't make a difference to have a CLA or not. What you trying to avoid is already allowed as per the license.
However, some CLAs require you to declare the originality and ownership of your contributions. Maybe that's why they require a CLA to be signed for contributions.
Therefore, it doesn't make a difference to have a CLA or not. What you trying to avoid is already allowed as per the license.
However, some CLAs require you to declare the originality and ownership of your contributions. Maybe that's why they require a CLA to be signed for contributions.
> How would you fund the open source development of a database?
I think the best way would be something similar to the Linux Foundation. Companies in need of a certain type of database would pool resources to develop and maintain it.
PostgreSQL was a community developed database that was funded in something approaching this way as most of the developers were either students or worked for someone who paid them while they worked on it.
> TiDB is Apache licensed, that should be enough no?
It's good enough to use it with the knowledge that it isn't truly free (libre) because the CLA gives the owning company the right to change that at any time. So you can use it but don't build a business around it or make it a critical part of your infrastructure as they might pull the rug out from under you.
I think the best way would be something similar to the Linux Foundation. Companies in need of a certain type of database would pool resources to develop and maintain it.
PostgreSQL was a community developed database that was funded in something approaching this way as most of the developers were either students or worked for someone who paid them while they worked on it.
> TiDB is Apache licensed, that should be enough no?
It's good enough to use it with the knowledge that it isn't truly free (libre) because the CLA gives the owning company the right to change that at any time. So you can use it but don't build a business around it or make it a critical part of your infrastructure as they might pull the rug out from under you.
TiKV the Core Storage scalable component is a CNCF graduated projected. PingCAP cannot change the license even if it wanted to.
Thanks for the info! I wasn't aware of this and didn't see any mention of this in their github docs. It means that at least you can use that part of the project without worry. They should promote this aspect of the project more prominently. It should be of great interest to anyone thinking about adopting the technology.
Do you have any links to the Core Storage CNCF project? I couldn't find it on the CNCF website under the Database and Cloud Native Storage sections?
Do you have any links to the Core Storage CNCF project? I couldn't find it on the CNCF website under the Database and Cloud Native Storage sections?
> the CLA gives the owning company the right to change that at any time
It isn't copyleft, so even without a CLA, they could make a proprietary or open core version of the product, as can anyone else.
It isn't copyleft, so even without a CLA, they could make a proprietary or open core version of the product, as can anyone else.
> I think the best way would be something similar to the Linux Foundation. Companies in need of a certain type of database would pool resources to develop and maintain it.
Disclaimer: I currently work at PingCAP and previously worked at the Linux Foundation.
Foundation (whether it's the LF or others) isn't a panacea either. It's fine when projects get started and there are plenty of willing member companies, but for many projects, companies often lose interest, need to cut down on open source related investments, etc. and the project funding dwindles. The reality is that the member companies need to both fund and provide software developers for projects, and it's difficult to expect them to keep the same commitment for more than a few years these days....
Disclaimer: I currently work at PingCAP and previously worked at the Linux Foundation.
Foundation (whether it's the LF or others) isn't a panacea either. It's fine when projects get started and there are plenty of willing member companies, but for many projects, companies often lose interest, need to cut down on open source related investments, etc. and the project funding dwindles. The reality is that the member companies need to both fund and provide software developers for projects, and it's difficult to expect them to keep the same commitment for more than a few years these days....
With it under the Apache 2 license, misuse of a CLA to get free contributions from the community that aren't shared back to the community isn't currently a problem.
Is your main issue with the VC funding, the CLA, or the combination of both. Doesn’t Kubernetes (hosted by the Linux Foundation) also have a CLA?
CLAs are not necessarily a bad thing. In fact, they are pretty much necessary for software companies. We have one for our open source projects. It has language to ensure there's no dispute about our right to the code. That's important, because we indemnify our customers against IP infringement. It's a standard part of contracts for software and services.
IANAL but you can have that without getting yourself a full copyright license to all contributions - as CLAs generally do, and what enables license changes.
A popular more lightweight solution is the DCO[0].
[0]: https://developercertificate.com/
A popular more lightweight solution is the DCO[0].
[0]: https://developercertificate.com/
I love these new distributed DBs. CockroachDB is one of them. Still I think a managed Postgres/MySQL is a better choice. My primary concern is how challenging will it be if you have to eventually move your data out to a RDBMS for cost or other reasons. Does anyone have any experience ? I am not talking enterprise scale but data about size of 50 - 100GB scale.
Distributed DBs and traditional RDBMS serve different purposes. Most – by which I mean the overwhelming majority – companies do not need a distributed DB, they need better schema and queries.
My fear is companies without in-house RDBMS expertise see these products as a way to continue to avoid getting that expertise.
My fear is companies without in-house RDBMS expertise see these products as a way to continue to avoid getting that expertise.
I think it is mostly three folds. Availability, to be cloud agnostic and scale of few tbs to few hundred TBs. I have been seeing cockroach and yugabyte in an on prem setup where you don't get the benefits of a cloud provider but you can get the availability guarantees and fault tolerance.
why do you think that would be harder? assume for the moment that the reader here is going to run at the effective rate of a single node and not that we're going to try to parallelize that. Assuming we have transaction isolation, that reader is going to get a consistent snapshot.
a distributed database is potentially more complicated to operate, and optimize, and because its new and potentially has more sharp edges maybe less reliable (?) but the extraction of moderate sized datasets doesn't really seem to be an obvious failing.
a distributed database is potentially more complicated to operate, and optimize, and because its new and potentially has more sharp edges maybe less reliable (?) but the extraction of moderate sized datasets doesn't really seem to be an obvious failing.
I would argue the opposite, distributed databases are much easier to operate at large scale. Truly online distributed DDL, at least in TIDB, strong consistency etc.
People who bang on about Postgres replication have rarely setup replication in Postgres themselves and that too in the 100a of Pb scale.
MySQL replication works well and can be scaled more easily (relative to Postgres) but has its own problems. eg., DDL is still a nightmare, lag is a real problem, usually masked by async replication. But then eventual consistency makes the application developers life more complicated.
People who bang on about Postgres replication have rarely setup replication in Postgres themselves and that too in the 100a of Pb scale.
MySQL replication works well and can be scaled more easily (relative to Postgres) but has its own problems. eg., DDL is still a nightmare, lag is a real problem, usually masked by async replication. But then eventual consistency makes the application developers life more complicated.
Interview with TiDB developer explaining some of the architecture of the DB : https://youtu.be/mmzoSkEhYrA?si=5eIppYzqKvOxWDtv
TiDB is 8 years old now; what's new?
Yeah that and the actual core storage engine is a key-value store library called https://github.com/tikv/tikv which is written in Rust. I suspect the Go part only handles the replication component, MVCC, query planner, server etc.
Is tikv (still) a wrapper of RocksDB?
Hardly hear about TiDB. How is it faring and how does it stack up to more well know competition?
Same as before, excellent when you little little data and limited transactions. Once things get big and you lose sync, everything goes to shit.
They claimed they successfully onboarded a customer with 1 PB of data at KubeCon NA.
I have strong doubts about distributed DBs in general, but I also can’t see them blatantly lying in a talk.
I have strong doubts about distributed DBs in general, but I also can’t see them blatantly lying in a talk.
how do you have doubts when nearly every significant company is running one, and distributed DBs are consistently proven to be the most correct (e.g. FDB, TigerBeetle)?
Sorry, should have clarified: I have doubts about their necessity (as I stated in another comment).
Most tech companies have poor knowledge of proper data modeling and SQL, leading to poor schema design, and suboptimal queries. Combine that with the fact that networked storage (e.g. EBS) is the norm, and it’s no wonder that people think they need another solution.
The amount of QPS you can get out of a single DB is staggering when it’s correctly designed, and on fast hardware with local NVMe disks (or has a faster distributed storage solution). Consider that a modern NVMe drive can quite easily deliver 1,000,000+ IOPS.
Most tech companies have poor knowledge of proper data modeling and SQL, leading to poor schema design, and suboptimal queries. Combine that with the fact that networked storage (e.g. EBS) is the norm, and it’s no wonder that people think they need another solution.
The amount of QPS you can get out of a single DB is staggering when it’s correctly designed, and on fast hardware with local NVMe disks (or has a faster distributed storage solution). Consider that a modern NVMe drive can quite easily deliver 1,000,000+ IOPS.
distributed DBs provide seamless fault tolerance story usually.
> Consider that a modern NVMe drive can quite easily deliver 1,000,000+ IOPS.
There could be other bottlenecks, for example I am consistently experiencing that linux kernel doesn't handle memory pages allocations fast enough once disk traffic hits few GB/s, because it does it in single thread.
> Consider that a modern NVMe drive can quite easily deliver 1,000,000+ IOPS.
There could be other bottlenecks, for example I am consistently experiencing that linux kernel doesn't handle memory pages allocations fast enough once disk traffic hits few GB/s, because it does it in single thread.
It all depends on what kind of queries you're running. I came from the OLTP market, where you're generally doing single-row operations. Basic CRUD. Single table work on denormalized data.
Now go to OLAP, and a single query might be doing multiple table joins. It might be scouring billions of records. It might need to do aggregations. Suddenly "millions of ops" might be reduced to 100 QPS. If you're lucky.
And yes, that's even using fast local NVMe. It's just a different kind of query, with a different kind of result set. YMMV.
Now go to OLAP, and a single query might be doing multiple table joins. It might be scouring billions of records. It might need to do aggregations. Suddenly "millions of ops" might be reduced to 100 QPS. If you're lucky.
And yes, that's even using fast local NVMe. It's just a different kind of query, with a different kind of result set. YMMV.
Not sure why you think OLTP doesn’t also do complex joins. In a properly normalized schema, you’ll likely have many.
But yes, OLAP is of course its own beast, and most DBs are suited for one or the other.
But yes, OLAP is of course its own beast, and most DBs are suited for one or the other.
I think it's a matter of use case. Doing ad hoc data exploration on an OLTP system generally sucks the wind out of the performance. Even if you have some type of workload prioritization, isolation, and limitation, allowing data scientists and business analysts freely wandering through your production OLTP database sounds like a Bad Time.
The organization might say "Okay. Maybe you should do your ad hoc exploration on an OLAP system. Preferably our data warehouse where you can let your report run for hours and we won't see a production brownout while it's running."
So complexity of ad hoc joins in the warehouse generally can get more complex.
The organization might say "Okay. Maybe you should do your ad hoc exploration on an OLAP system. Preferably our data warehouse where you can let your report run for hours and we won't see a production brownout while it's running."
So complexity of ad hoc joins in the warehouse generally can get more complex.
"every significant company"? i'd say most of them just use postgres or mysql and scale fine.
If you look at DB-engines.com/ranking and look at all of the collective interest in all of the databases listed, you will see that the aggregate "score" of all databases combined 7105.84. Postgres is indeed popular; but it is only ranked 4th on the list, with its own score of 648.96. MySQL currently is still 50% larger in terms of interest, with a score of 998.15.
Which means interest in Postgres (specifically) is only 9.13% of overall interest in databases; MySQL another 14.04%. Combined 23.27%.
Is that a significant percentage of interest? Yes. Many others are a fraction of 1% of mindshare in the market.
Yet the reason there are 423 systems ranked in DB-Engines is because no one size fits all data, or data query patterns, or workloads, or SLAs, or use cases.
PostgreSQL and MySQL are, at the end of the day, oriented towards OLTP workloads. While you can stretch them to be used for OLAP, these are "unnatural acts." They were both designed in days long ago for far smaller datasets than typical for modern-day petabyte-scale, real-time (streaming) ingestion, cloud-native deployments. While many engineering teams have cobbled together PostgreSQL and MySQL frankenservers designed for petabyte-scale workloads, YMMV for your data ingest, and for p99s and QPS.
The dynamic at play here is that there are some projects that lend themselves to "general services" databases, where MySQL or PostgreSQL or anything else to hand is useful for them. And then there are specialized databases designed for purpose for certain types of workloads, data models, query patterns, use cases, and so on.
So long as "chaos" fights against "law" in the universe, you will see this desire to have "one" database standard rule them all, versus a Cambrian explosion of options for users and use cases.
Which means interest in Postgres (specifically) is only 9.13% of overall interest in databases; MySQL another 14.04%. Combined 23.27%.
Is that a significant percentage of interest? Yes. Many others are a fraction of 1% of mindshare in the market.
Yet the reason there are 423 systems ranked in DB-Engines is because no one size fits all data, or data query patterns, or workloads, or SLAs, or use cases.
PostgreSQL and MySQL are, at the end of the day, oriented towards OLTP workloads. While you can stretch them to be used for OLAP, these are "unnatural acts." They were both designed in days long ago for far smaller datasets than typical for modern-day petabyte-scale, real-time (streaming) ingestion, cloud-native deployments. While many engineering teams have cobbled together PostgreSQL and MySQL frankenservers designed for petabyte-scale workloads, YMMV for your data ingest, and for p99s and QPS.
The dynamic at play here is that there are some projects that lend themselves to "general services" databases, where MySQL or PostgreSQL or anything else to hand is useful for them. And then there are specialized databases designed for purpose for certain types of workloads, data models, query patterns, use cases, and so on.
So long as "chaos" fights against "law" in the universe, you will see this desire to have "one" database standard rule them all, versus a Cambrian explosion of options for users and use cases.
While you’re not wrong re: Postgres and MySQL not being necessarily designed for PB-scale, IME many shops with huge datasets are just doing it wrong. Storing numbers as strings, not using lookup tables for low-cardinality data, massive JSON blobs everywhere, etc.
I’m not saying it fixes everything, but knowing how a DB works, and applying proper data modeling and normalization could severely reduce the size of many datasets.
I’m not saying it fixes everything, but knowing how a DB works, and applying proper data modeling and normalization could severely reduce the size of many datasets.
Yes. The chance to do wrong things with data increases with scale.
TiDB is running core banking services too.
I think people’s idea of scale and operating at scale is limited to their experience.
You can get MySQL to run at any scale, look at Meta and Shopify. Operational complexity at that scale is a different story.
Distributed databases reduce a lot of the operational complexity. To take one example:
Try a DDL on a 5 TB table in any replicated MySQL topology of choice and compare it with TiDB’s Distributed execution framework.
I think people’s idea of scale and operating at scale is limited to their experience.
You can get MySQL to run at any scale, look at Meta and Shopify. Operational complexity at that scale is a different story.
Distributed databases reduce a lot of the operational complexity. To take one example:
Try a DDL on a 5 TB table in any replicated MySQL topology of choice and compare it with TiDB’s Distributed execution framework.
> Distributed databases reduce a lot of the operational complexity
That’s a new one to me; never have I ever heard someone claim that making a system distributed reduces complexity.
I’ve operated EDB’s Distributed Postgres at a decent scale (< 100 TB of unique data); in no way did it enable reduced complexity. For the devs, maybe – “chuck whatever data you want wherever you want, it’ll show up.” But for Ops? Absolutely not, it was a nightmare.
That’s a new one to me; never have I ever heard someone claim that making a system distributed reduces complexity.
I’ve operated EDB’s Distributed Postgres at a decent scale (< 100 TB of unique data); in no way did it enable reduced complexity. For the devs, maybe – “chuck whatever data you want wherever you want, it’ll show up.” But for Ops? Absolutely not, it was a nightmare.
Note that TiDB did subject itself to Jepsen testing (relatively) early. Here's their 2019 results:
https://jepsen.io/analyses/tidb-2.1.7
The devil is in the details, and anyone who is looking to implement TiDB for data correctness should read through not just this but other currently-open correctness-related Github issues:
e.g., https://github.com/pingcap/tidb/issues?q=is%3Aissue%20state%...
TiDB currently has 74 open issues related to "correctness," and has closed 163.
https://github.com/pingcap/tidb
https://jepsen.io/analyses/tidb-2.1.7
The devil is in the details, and anyone who is looking to implement TiDB for data correctness should read through not just this but other currently-open correctness-related Github issues:
e.g., https://github.com/pingcap/tidb/issues?q=is%3Aissue%20state%...
TiDB currently has 74 open issues related to "correctness," and has closed 163.
https://github.com/pingcap/tidb
This is quite an ignorant comment. TiDB routinely handles 100s of TB of data.
Go watch LinkedIn’s presentation on why they choose TiDB as their strategic db going forward.
Do you have a link?
Edit: https://www.youtube.com/watch?v=DzZ-I8WL2jM
Edit: https://www.youtube.com/watch?v=DzZ-I8WL2jM
I hear through the grapevine they’ve got a bunch of companies using it. I think Airbnb migrated to it.
There are quite a few. Pinterest, LinkedIn, Plaid, Mercari, and Rakuten shared their experiences with TiDB at PingCAP's annual event last year (2024) - https://www.pingcap.com/htap-summit/.
In the previous event (2023) there were speakers from Airbnb, Databricks, Flipkart, PayPay, and others sharing their experiences as well - https://www.pingcap.com/htap-summit/sept-2023/
Disclosure: Employee of PingCAP the company behind TiDB
In the previous event (2023) there were speakers from Airbnb, Databricks, Flipkart, PayPay, and others sharing their experiences as well - https://www.pingcap.com/htap-summit/sept-2023/
Disclosure: Employee of PingCAP the company behind TiDB
> PingCAP's annual event last year (2014)
Wasn't last year 2024?
Wasn't last year 2024?
Yes, I am still getting up to speed after a long break for Christmas and New Year!
Not only are these companies using TiDB, they're also contributing to it as you can see in this recent blog post: https://medium.com/@ray.paik/tidb-8-5-is-here-b2ca09e972d5
[deleted]
I was able to get a small TiDB cluster running in my homelab last night! Looking forward to using it :)
Reference: https://docs.pingcap.com/tidb-in-kubernetes/stable/get-start...
Reference: https://docs.pingcap.com/tidb-in-kubernetes/stable/get-start...
What is that first paragraph? "This implementation proves the CAP theorem as wrong"? Doesn't tell me what your drawbacks are. This is mongodb all over again. Will only ever consider if the customer really wants it. (My guess performance)
(Also, bold move to write a db without fully being able to manage memory).
(Also, bold move to write a db without fully being able to manage memory).
I do want to clarify a few points, on the project page it does provide the following information:
> Distributed Transactions: TiDB uses a two-phase commit protocol to ensure ACID compliance, providing strong consistency. Transactions span multiple nodes, and TiDB's distributed nature ensures data correctness even in the presence of network partitions or node failures.
> High Availability: Built-in Raft consensus protocol ensures reliability and automated failover. Data is stored in multiple replicas, and transactions are committed only after writing to the majority of replicas, guaranteeing strong consistency and availability, even if some replicas fail. Geographic placement of replicas can be configured for different disaster tolerance levels.
See https://github.com/pingcap/tidb?tab=readme-ov-file#key-featu...
Correctness has been a focus for a long time for TiDB, including working on passing Jepsen Tests back in 2019, see https://www.pingcap.com/blog/tidb-passes-jepsen-test-for-sna... and https://jepsen.io/analyses/tidb-2.1.7
Disclosure: Employee of PingCAP the company behind TiDB
> Distributed Transactions: TiDB uses a two-phase commit protocol to ensure ACID compliance, providing strong consistency. Transactions span multiple nodes, and TiDB's distributed nature ensures data correctness even in the presence of network partitions or node failures.
> High Availability: Built-in Raft consensus protocol ensures reliability and automated failover. Data is stored in multiple replicas, and transactions are committed only after writing to the majority of replicas, guaranteeing strong consistency and availability, even if some replicas fail. Geographic placement of replicas can be configured for different disaster tolerance levels.
See https://github.com/pingcap/tidb?tab=readme-ov-file#key-featu...
Correctness has been a focus for a long time for TiDB, including working on passing Jepsen Tests back in 2019, see https://www.pingcap.com/blog/tidb-passes-jepsen-test-for-sna... and https://jepsen.io/analyses/tidb-2.1.7
Disclosure: Employee of PingCAP the company behind TiDB
I can't find that first paragraph, and it seems that readme file was last modified 3 months ago, though you're using quotes, so perhaps you are paraphrasing?
The first paragraph I find under Key Features is:
Distributed Transactions: TiDB uses a two-phase commit protocol to ensure ACID compliance, providing strong consistency. Transactions span multiple nodes, and TiDB's distributed nature ensures data correctness even in the presence of network partitions or node failures.
It claims to ensure correctness in the presence of partitions, not availability, so I don't think that's a claim that CAP is wrong? I would expect it to be unavailable if it can't provide a consistent response.
I don't know about memory management. It seems go may not be used for the storage layer, but in any case, it wouldn't be the first attempt at a DB on a garbage collected runtime. I can think of at least Cassandra on the jvm and CockroachDB also on Go. I do prefer that DBs do their own memory management (as well as storage) but this is not as unique as it used to be though.
The first paragraph I find under Key Features is:
Distributed Transactions: TiDB uses a two-phase commit protocol to ensure ACID compliance, providing strong consistency. Transactions span multiple nodes, and TiDB's distributed nature ensures data correctness even in the presence of network partitions or node failures.
It claims to ensure correctness in the presence of partitions, not availability, so I don't think that's a claim that CAP is wrong? I would expect it to be unavailable if it can't provide a consistent response.
I don't know about memory management. It seems go may not be used for the storage layer, but in any case, it wouldn't be the first attempt at a DB on a garbage collected runtime. I can think of at least Cassandra on the jvm and CockroachDB also on Go. I do prefer that DBs do their own memory management (as well as storage) but this is not as unique as it used to be though.
Go is used for the SQL layer, it’s a modern optimizer that can do distributed joins etc. ie. Issue parallel reads to the storage nodes.
Additionally, it can push down the DAG to the TiKV storage nodes, written in Rust, to reduce movement of data and work closer to the physical data.
Additionally, it can push down the DAG to the TiKV storage nodes, written in Rust, to reduce movement of data and work closer to the physical data.
Not having a single binary distribution like CockroachDB is hurting adoption of TiDB.
Have you looked at tiup?
https://tiup.io
https://tiup.io
Does "cloud native" mean "has kubernetes config" or is there something else to it?
1. SQL front end nodes 2. Distributed shared nothing storage (TiKV) 3. Meta data server (PS) 4. TiFlash column store
1 and 3 are written in Go 2 is written in Rust and uses RocksDB 4 is written in C++
2 & 3 are graduated CNCF projects maintained by PingCAP.
Disclaimer: I work for PingCAP