1/4. The max number of recipients of any prize is 3, but it can be split into a half, and then one of those halves into two further halves, for a 1/2 + 1/4 + 1/4 split. (It can also be split equally into 1/3 + 1/3 + 1/3).
you can get expected "single shard" performance in CockroachDB by manually splitting the shards (called "ranges" in CockroachDB) along the lines of the expected single shard queries (what you call a "properly shared database"). This is easy to do with a single SQL command. (This is what we do today; we use CockroachDB for strongly consistent metadata).
The difference between CockroachDB and a manually sharded database is that when you _do_ have to perform some cross-shard transactions (which you inevitably have to do at some point), in CockroachDB you can execute them (with a reasonable performance penalty) with strong consistency and 2PC between the shards, whereas in your manually sharded database... good luck! Hope you implement 2PC correctly.
This disconnect is called the "Tullock Paradox" in Economics, after Gordon Tullock who first asked it in "The Purchase of Politicians" (1972) [couldn't find an online link].
You'll find a more recent discussion in "Why is There so Little Money in U.S. Politics?" (2003). [1]
Materialize | Engineering, Product, Marketing | NYC HQ + North America Remote + Europe Remote | https://materialize.com/careers
Materialize is a streaming database for real-time applications. Materialize lets you ask questions about your data, and then get low-latency, correct answers, which are kept incrementally updated as the underlying data changes.
Materialize is built on Timely Dataflow, a low-latency cyclic dataflow computational model, first introduced in the paper "Naiad: a timely dataflow system".
Materialize is a team of over thirty, primarily based in New York City but also open to remote positions in the EU and NA. We are hiring in all engineering positions (eng. manager, engineers from new grad to principal) as well as several non-engineering positions - for the full list, see https://materialize.com/careers
We are a team of significantly experienced individuals in databases and distributed systems, and looking to add more folks with that interest and/or experience to our team. Materialize recently raised a $32m Series B led by Kleiner Perkins, which was lovingly hacker newsed: https://news.ycombinator.com/item?id=25277511
A high speed collision in low orbit can change a circular low orbit into an elliptical eccentric orbit that intersects a higher circular orbit, but unless there is an additional accelerating event at that higher altitude, it cannot recircularize its orbit at that higher altitude.
There are thus two takeaways:
1. By definition, this means that part of the orbit will always be at low altitude, regardless of the collision dynamics. So this means that it will continue to decay over time, albeit perhaps at a slower rate (decay being proportional to the time spent at lower altitude).
2. While that eccentric orbit will intersect with a higher circular orbital plane, it does so in a predictable fashion that can be routed around. The higher orbits are also much sparser, so the chance of this intersecting with a satellite that is already present is very, very small.
Materialize | Engineering, Product, Marketing | NYC HQ + North America Remote + Europe Remote | https://materialize.com/careers
Materialize is a streaming database for real-time applications. Materialize lets you ask questions about your data, and then get low-latency, correct answers, which are kept incrementally updated as the underlying data changes.
Materialize is built on Timely Dataflow, a low-latency cyclic dataflow computational model, first introduced in the paper "Naiad: a timely dataflow system".
Materialize is a team of over thirty, primarily based in New York City but also open to remote positions in the EU and NA. We are hiring in all engineering positions (eng. manager, engineers from new grad to principal) as well as several non-engineering positions - for the full list, see https://materialize.com/careers
We are a team of significantly experienced individuals in databases and distributed systems, and looking to add more folks with that interest and/or experience to our team. Materialize recently raised a $32m Series B led by Kleiner Perkins, which was lovingly hacker newsed: https://news.ycombinator.com/item?id=25277511
Thank you very much for the elaboration, I really appreciate the thinking!
> Personally, I'm less interested in one-off migrations like you're suggesting. What I really want is to have something like Materialize embedded in my Postgres.
We're about to launch "Materialize as a Postgres read-replica" where it connects to a Postgres leader just as a Postgres read-replica would - using the built-in streaming replication in newer versions of postgres. It's currently in final testing before being released in the next month or two.
Also on our roadmap for Materialize Cloud is plug-and-play connections with Fivetran, Hightouch, and Census (and more) to bring in the business data, and allow you to, as you put it, make Materialize the central collection point for keeping updated all your views.
Thank you for your kind words! We indeed have plenty of work to be done (and are thus hiring)! I'm curious however why you think this requires you to be all-in on Materialize. As you said better than I could have, dbt is amazing at keeping your business logic organized. Our intention is very much for dbt to standardize the modeling/business logic layer which allows you to use multiple backends as you see fit in a way that shares the catalog layer cleanly.
Our hope is that you have some BigQuery/Snowflake job that you're tired of running up the bill hitting redeploy 5 times a day, and you can cleanly port that over to Materialize with little work because the adapter is taking care of any small semantic differences in date handling, or null handling, etc. So Materialize sits cleanly side-by-side with Snowflake/BigQuery, and you're choosing whether you want things incrementally maintained with a few seconds of latency by Materialize, or once a day by the batch systems.
My view is you're likely going to want to do data science with a batch system (when you're in "learning mode" you try and keep as many things fixed, including not updating the dataset), and then if the model becomes a critical automated pipeline, rather than rerunning the model every hour and uploading results to a Redis cache or something, you switch it over to Materialize, and don't have to every worry about cache invalidation.
Materialize | Engineering, Product, Marketing | NYC HQ + North America Remote + Europe Remote | https://materialize.com/careers
Materialize is a streaming database for real-time applications. Materialize lets you ask questions about your data, and then get low-latency, correct answers, which are kept incrementally updated as the underlying data changes.
Materialize is built on Timely Dataflow, a low-latency cyclic dataflow computational model, first introduced in the paper "Naiad: a timely dataflow system".
Materialize is a team of around thirty, primarily based in New York City but also open to remote positions in the EU and NA. We are hiring in all engineering positions (eng. manager, engineers from new grad to principal) as well as several non-engineering positions - for the full list, see https://materialize.com/careers
We are a team of significantly experienced individuals in databases and distributed systems, and looking to add more folks with that interest and/or experience to our team. Materialize recently raised a $32m Series B led by Kleiner Perkins, which was lovingly hacker newsed: https://news.ycombinator.com/item?id=25277511
protobufs still get encoded and decoded by each client when loaded into memory. arrow is a little bit more like "flatbuffers, but designed for common data-intensive columnar access patterns"
Materialize | Engineering, Product, Marketing | NYC HQ + North America Remote + EU Remote | http://materialize.io/careers
Materialize is a streaming database for real-time applications. Materialize lets you ask questions about your data, and then get low-latency, correct answers, which are kept incrementally updated as the underlying data changes.
Materialize is built on Timely Dataflow, a low-latency cyclic dataflow computational model, first introduced in the paper "Naiad: a timely dataflow system".
Materialize is a team of around thirty, primarily based in New York City but also open to remote positions in the EU and NA. We are hiring in all engineering positions (eng. manager, engineers from new grad to principal) as well as several non-engineering positions - for the full list, see http://materialize.io/careers
We are a team of significantly experienced individuals in databases and distributed systems, and looking to add more folks with that interest and/or experience to our team. Materialize recently raised a $32m Series B led by Kleiner Perkins, which was lovingly hacker newsed: https://news.ycombinator.com/item?id=25277511
Hi! I'm one of the two authors here. At Materialize, we're definitely of the 'we are a bunch of voices, we are people rather than corp-speak, and you get our largely unfiltered takes' flavor. This is my (and George's from Fivetran) take. In particular this is not Frank's take, as you attribute below :)
> SQL is declarative, reactive Materialize streams are declarative on a whole new level.
Thank you for the kind words about our tech, I'm flattered! That said, this dream is downstream of Kafka. Most of our quibbles with the Kafka-as-database architecture are to do with the fact that that architecture neglects the work that needs to be done _upstream_ of Kafka.
That work is best done with an OLTP database. Funnily enough, neither of us are building OLTP databases, but this piece largely is a defense of OLTP databases (if you're curious, yes, I'd recommend CockroachDB), and their virtues at that head of the data pipeline.
Kafka has its place - and when its used downstream of CDC from said OLTP database (using, e.g. Debezium), we could not be happier with it (and we say so).
The best example is in foreign key checks. It is not good if you ever need to enforce foreign key checks (which translates to checking a denormalization of your source data _transactionally_ with deciding whether to admit or deny an event). This is something that you may not need in your data pipeline on day 1, but adding that in later is a trivial schema change with an OLTP database, and exceedingly difficult with a Kafka-based event sourced architecture.
> Normally you'd have single writer instances that are locked to the corresponding Kafka partition, which ensure strong transactional guarantees, IF you need them.
This still does not deal with the use-case of needing to add a foreign key check. You'd have to:
1. Log "intents to write" rather than writes themselves in Topic A
2. Have a separate denormalization computed and kept in a separate Topic B, which can be read from. This denormalization needs to be read until the intent propagates from Topic A.
3. Convert those intents into commits.
4. Deal with all the failure cases in a distributed system, e.g. cleaning up abandoned intents, etc.
If you use an OLTP database, and generate events into Kafka via CDC, you get the best of both worlds. And hopefully, yes, have a reactive declarative stack downstream of that as well!
If you're interested in a job, apply and talk to us, that's by far the best way! If you just want to chat - my profile has a variety of ways to contact me as well.
Materialize | Engineers | Marketing | NYC HQ + North America Remote + EU Remote in early 2021 | http://materialize.io/careers
Materialize is a streaming database for real-time applications. Materialize lets you ask questions about your data, and then get low-latency, correct answers, which are kept incrementally updated as the underlying data changes.
Materialize is built on Timely Dataflow, a low-latency cyclic dataflow computational model, first introduced in the paper "Naiad: a timely dataflow system". Materialize is co-founded by Frank McSherry, the primary author of Timely Dataflow (http://timelydataflow.com) and Differential Dataflow (http://differentialdataflow.com), the two open source projects that power Materialize. Materialize itself is source-available and entirely written in Rust: https://github.com/MaterializeInc/materialize
Materialize is a team of over twenty, primarily based in New York City but also open to remote positions. We are hiring in all engineering positions (eng. manager, engineers from new grad to principal) as well as several non-engineering positions. For the full list, see http://materialize.io/careers
We are a team of significantly experienced individuals in databases and distributed systems, and looking to add more folks with that interest and/or experience to our team.
The reasoning is that as you gain information, you also have a duty to the people in the control group to use the best available information to take care of their health. Once you gain "enough" information ("enough" being statistically defined) that the drug helps, each additional person you let languish in the control group (who is denied access to the drug) is a cost that must be weighed against the benefit of getting additional information. When the data is clear enough, the cost can exceed the benefit, and you stop early.
Typically you'd register a "stopping rule" before you start your trial: a good drug often will trigger the stopping rule, as it helped so much that we learned about its efficacy on a smaller N than originally planned.
There are many different stopping criteria, depending on the trial (in safety trials you'd typically stop because you've found evidence that the drug is unsafe and continuing would be unfair to the folks in the treatment group, whereas in efficacy trials after safety has been established, you'd stop because you've found evidence that the drug is effective and continuing would be unfair to the folks in the control group).
Materialize is a streaming database for real-time applications. Materialize lets you ask questions about your data, and then get low-latency, correct answers, which are kept incrementally updated as the underlying data changes.
Materialize is built on Timely Dataflow, a low-latency cyclic dataflow computational model, first introduced in the paper "Naiad: a timely dataflow system". Materialize is co-founded by Frank McSherry, the primary author of Timely Dataflow (http://timelydataflow.com) and Differential Dataflow (http://differentialdataflow.com), the two open source projects that power Materialize. Materialize itself is source-available and entirely written in Rust: https://github.com/MaterializeInc/materialize
Materialize is a team of over twenty, primarily based in New York City but also open to remote positions. We are hiring in all engineering positions (eng. manager, engineers from new grad to principal) as well as several non-engineering positions. For the full list, see http://materialize.io/careers
We are a team of significantly experienced individuals in databases and distributed systems, and looking to add more folks with that interest and/or experience to our team.
Yes, we do! It's a little non-standard SQL syntax we added called TAIL. You write TAIL <viewname>, and you get changes pushed to you. You can see a video of me badly explaining it in [2]. You can also do the thing where you create a Kafka SINK of a view and have the system push this to Kafka rather than have a long-running open SQL connection. There's some fun stuff with TAIL AS OF (start the changes at a specific point in time) and TAIL WITH SNAPSHOT (run a SELECT query, and then begin the change stream transactionally with the time of that SELECT query) that you can read about in the docs[1].
I don't believe Metabase supports the Calcite SQL dialect (https://github.com/metabase/metabase/issues/6230), which is what Crux is using for the SQL layer. So I believe the answer is no - but I'm not an expert here, so don't take this answer as definitive.
The argument is that without stronger consistency guarantees you can't do joins between two streams (or even something like argmax over a single stream, since it splits the stream into two subcomputations, which then have to be joined back together).
I think when folks say that eventual consistency is okay, they're thinking about simple aggregates - where transient incorrectness in the result is indistinguishable from noise.
But if you want to do joins, you really want to be able to reason about your unbounded streams causally - Flink, Beam, (and as another commenter points out, Firebase as well) provide stronger consistency guarantees on computations over unbounded streams.