One peice of tooling I've wanted for Vitess is for it to make recommendations about sharding keys based on the statistics it collects + query patterns. Seems like it's in the perfect location to collect the data for that and remove the mental overhead associated w sharding. If much of the implementation has to be rebuilt, is there a path for that in this implementation?
OrioleDB maintains a very small set of Postgres patches that are targeted for upstream. The storage engine that mitigates the need for vacuuming [1] is implemented in a standard Postgres extension, so that will still need to be installed by the Postgres host in order to take advantage of it.
I don't want to hijack Datadogs+Quickwit's post comment section with unrelated promotional-looking info. Quick summary below but if you have any other questions pls tag olirice in a Supabase GH discussion.
The OrioleDB storage engine for postgres is a drop-in replacement for the default heap method. Its takes advantage of modern hardware (e.g. SSDs) and cloud infrastructure. The most basic benefit is that throughput at scale is > 5x higher than heap [1], but it also is architected for a bunch of other cool stuff [2]. copy-on-write unblocks branching. row-level-WAL enables an S3 backend and scale-to-zero compute. The combination of those two makes it a suitable target for multi-master.
So yes, given that it could greatly improve performance on the platform, it is a goal to release in Supabase's primary image once everything is buttoned up. Note that an OrioleDB release doesn't take away any of your existing options. Its implemented as an extension so users would be able to optionally create all heap tables, all orioledb tables, or a mix of both.
OrioleDB continues to be a fully open source and liberally licensed. We're working with the OrioleDB team to provide an initial distribution channel so they can focus on the storage engine vs hosting + providing lots of user feedback/bug reports. Our shared goal is to advance OrioleDB until it becomes the go-to storage engine for Postgres, both on Supabase and everywhere else.
yeah, fair shout
`columnar index` for hybrid analytical workloads and `multi-master` roadmap items haven't been put in an order on the schedule yet so maybe multi-master makes sense to do first. Both align with webscale requirements though. Columnar would move into the AlloyDB space and multi-master would be more like PlanetScale (although not quite the same audience in the latter case)
what's your use-case for multi-master that would not be supported by something like regional routed for read-replicas with high availability? i.e. do you have a specific need for low global write latency, or is it motivated by something else?
In Studio you'd need to click on a query to get the recommendations but the most important queries to optimize are on several metrics like "Most time consuming", "Most frequent", "Slowest execution" are sorted worst-to-best in tab views.
But if you do want to scan all of your queries at once for possible indexes you can do it in SQL with
Supabase is one of the most portable platforms out there.
The whole stack is self-hostable and open source. All of the data are contained in Postgres. You're one pg_dump away from being able to switch to a different Postgres host. Or if you're switching to something else entirely, you can export the data to CSVs and take it anywhere. But we're confident you won't want to :)
Exposing postgres functions through the GraphQL API has been one of our most frequent requests. I'm really looking forward to seeing what people are able to build with this flexibility. The coolest one I've come across so far is a full text search entrypoint on the Query object.
Really looking forward to the next steps with vecs and would love to hear any thoughts about the `future ideas`.
Mainly, what kind of interface would you like to see for adapters/transforms so collections can feel like you're inserting text/images/video/whatever rather than passing vectors around?
PostGIS is a native extension mostly written in C. Since there is no C trusted language it would be hard to fit PostGIS into the TLE paradigm
Its less of a problem for well known and trusted extensions like PostGIS though because it comes pre-installed on most hosted providers (Supabase, RDS, etc)
yeah, exactly. pgxn also houses native extensions, some of which may be TLE compatible. database.dev is trusted language extension specific
We've reached out to David, who runs pgxn to see how we might collaborate and allow pgxn to be used as another registry that users can install TLEs from
RDS does have the underlying native extension that is required to install user defined packages (pg_tle) from database.dev. It doesn't currently have support for the extension that is necessary to make the in-database client work to install packages directly from the registry.
That's great feedback though: We'll add a new page to the website for each package that provides the snippet you can run on an RDS instance to load the extension and then run `create extension`.
We're also in talks with RDS team and experimenting with a few options that would allow users to query the registry directly. The most promising one we have found so far is using AWS Lambda with RDS's aws_lambda extension to do that http part, and pass the contents back to the database. Still a WIP though!
index_advisor is a project that almost made the cut for LW7. We decided to launch it as a pgtle instead to get some feedback before we (spoiler) probably release it in LW8
It uses the hypopg package to very quickly generate an index for each column referenced in the query and then re-run's an explain plan to measure the estimated "total cost".
If the expected "total_cost" reduces, we add the indexes to the list of recommendations and continue testing the remaining columns.
Its currently limited to single column indexes only but in a later update we'll enhance that to do more comprehensive searches with multiple columns in a single index
> What should AWS do -> provide Postgres or similar as a database choice
If you're interested a postgres option you could check out Supabase GraphQL[1] which is based on pg_graphql [2], a native PostgreSQL extension.
You define your schema in SQL (including any indexes you'd like) and it reflects a full GraphQL API. With that stack, the example you gave about filtering for security would use be solved using a Row Level Security [3] policy where the work all occurs on the DB.
While not a part of pg_graphql, using Supabase as a backend also handles Auth [4] & Storage [5] (both OSS) which covers most of the Amplify bases
The 255 max length is something you could consider if untrusted people are able to insert data (through an API for example). A column in postgres can store up to 1 GB of data so its a possible DOS vector. Probably not an issue for most use-cases, just something that was considered in the application those code samples came from