lizztheblizz

il y a 5 ansdiscuss

I don't have nearly the experience in Postgres environments to have seen the same level of real-world impact there, but a quick search presents me with the following documentation, which seems to indicate mostly similar performance challenges related to the use of Foreign Key Constraints: https://www.postgresql.org/docs/13/populate.html#POPULATE-RM...

lizztheblizz

il y a 5 ansdiscuss

Depends. Have any examples of "normal" sharded MySQL databases?

EDIT: To clarify, sharding is not a standard feature included in Community Edition MySQL. Over the years, there have been various Oracle-initiated attempts at providing it as an enterprise scaling strategy through MySQL (NDB) Cluster, MySQL Fabric, etc., but these have either ended up having limited applicability outside very specific use cases and are not widely in use.

Most large MySQL users (e.g. Facebook or YouTube) ended up rolling their own frameworks, like Vitess, which has since been open sourced and adapted to more diverse environments. Until that became more accessible, though, the rest of the world mostly made do with wobbly multi-master setups relying on circular replication, behind some kind of proxy, or had to implement the sharding logic itself into their application code.

lizztheblizz

il y a 5 ansdiscuss

Give it a shot, because the current implementation specifically side steps the need for that full copy, and does let you test functionally against fully up-to-date production data.

You're right, though, it doesn't cover all of those uses cases. Luckily, Vitess does offer most of that out of the box already. Just need them exposed through the PlanetScale UI. :)

lizztheblizz

il y a 5 ansdiscuss

12+ years of massive scale production use for Vitess and 26+ years of hardening for MySQL and InnoDB. PlanetScale adds some (imho) great features on top of that, but it's standing on the shoulders of giants that have proven themselves over and over.

... Also, I guess it's MySQL-compatible rather than Postgres-compatible? :)

lizztheblizz

il y a 5 ansdiscuss

Already on it. :) https://vitess.io/docs/reference/vreplication/vreplication/

lizztheblizz

il y a 5 ansdiscuss

To be clear, this is not a Vitess/PlanetScale-specific opinion or choice. Foreign key constraints are a bit of a controversial topic in large-scale MySQL environments in general, which is the greater context in which this design decision was made by the Vitess team.

PlanetScale's (and Vitess') non-blocking schema changes rely on open source tools for MySQL like pt-online-schema-change and gh-ost, which are widely used in production environments everywhere, and neither of them are too comfortable supporting FK's, though pt-osc does accommodate them to some extent (https://www.percona.com/doc/percona-toolkit/3.0/pt-online-sc...). gh-ost's lack of support was discussed on HN previously here: https://news.ycombinator.com/item?id=16983620

A good collection of resources on why they're considered problematic and many companies designing large-scale MySQL schemas tend to drop them can also be found here: https://federico-razzoli.com/foreign-key-bugs-in-mysql-and-m...

lizztheblizz

il y a 5 ansdiscuss

Vitess' compatibility with MySQL has made major leaps in the past couple of versions and the team has started focusing on locking in ongoing compatibility with various popular development frameworks. You can find those here, and more are getting added regularly: https://github.com/planetscale/vitess-framework-testing/

The basics of MySQL compatibility are described in here, though it's important to keep in mind that just because something "works" doesn't always mean it's the best way to do things in a sharded environment: https://vitess.io/docs/reference/compatibility/mysql-compati...

lizztheblizz

il y a 5 ansdiscuss

That is exactly what we _do_ support. Our team includes the original developer of gh-ost for MySQL, which has been built to execute these kinds of massive changes at scale at GitHub. We've integrated it tightly with Vitess. Once your branch is ready to be merged with production, it executes that change in a completely non-blocking way.

lizztheblizz

il y a 5 ansdiscuss

The "ease" we used to refer to in Vitess primarily relates to its interaction with the application side, where it basically presents itself as "one big MySQL datastore". It uses a standard MySQL connector, and in general, once you have the infrastructure up and running with a compatible schema design, there's not too much to worry about from a coding standpoint. Sharding happens transparently to the application code, which generally translates to fewer code changes required.

Admittedly, that view left out the considerable challenge of actually deploying and running the infrastructure, designing and optimizing that schema, along with all the joys of managing large cluster environments.

That's what PlanetScale, the product, aims to solve. Dealing with clustering infrastructure IS a hurdle for most teams to overcome, and though Vitess' feature set and compatibility has expanded greatly to accommodate some of the most demanding use cases on the web today, a lot of its functionality can still be out of reach for a developer just trying to merge some code and a schema change.

Abstracting as much of that complexity away from the end user is the goal, as well as making their lives easier with a ton of the functionality we've always wanted to see built with Vitess. I can confirm that that is not an "easy" job on our end. :)

lizztheblizz

il y a 5 ansdiscuss

It's closer to your original assumption than your second. Branching relies on vreplication, and it does actually allow you to develop multiple versions of your schema against your full production dataset. The "magic" is a powerful materialization engine that allows every branch to maintain its own version, and it _does_ allow you to work with your actual data. This is just scratching the surface of what that can do, though. We have loads more features cooking. :)

lizztheblizz

il y a 5 ansdiscuss

This is monthly. :)

comments