- Materialize
- Flink SQL
- Arroyo
- Readyset
- RisingWave
- Timeplus
- Pathway
- Dozer
- ReadySet
- Snowflake dynamic tables
- Native materialized views in OLTP databases
- Just having a stack of views in your db
- Poor man's MVs with triggers
All subtly different on every spectrum from consistency, UDF support, operator support, latency, scaling/state limits, source/sink integrations, and compatibility with existing protocols.
What seems unique is the focus on "writebacks to the source without Kafka/Connect in between", instead of having either a built-in cache, serving as a stream processor, or both. It looks like the built-in cache is still available through the FDW deployment pattern.
They note that relative to the source tables they are eventually consistent (of course, unless you want to delay transaction writes) but it's not clear what other consistency aspects they respect (such as preserving transactions end to end).
Overall this looks like it's designed to overcome materialized view limitations (which in popular OLTP dbs are pretty severe w.r.t. either what operations are supported, latency, or both) compared to other solutions that basically move the action downstream...curious if it will see much use, or if they'll inevitably introduce sinks and direct access to see if they can compete in the "live ODS" segment with Materialize and RisingWave.
edit: to make my comment more clear: this is a new entrant in a crowded space with several sophisticated, established players and the main differentiation is the deployment pattern. I'd be curious to know if anything else sets them apart
Agree with this. Snowflake has best-in-class dev experience and performance for Spark-like workloads (so ETL or unconstrained analytics queries).
It has close to worst-in-class performance as a serving layer.
If you're creating an environment to serve analysts and cached BI tools, you'll have a great time.
If you're trying to drive anything from Snowflake where you care about operations measured in ms or single digit seconds, you'll have a bad time and probably set a lot of money on fire in the process.
If you want an Airflow-ish approach without punishing your future self, pick Prefect. Otherwise go with Temporal. Above all do not adopt Airflow for the use cases you describe in 2023
- Materialize - Flink SQL - Arroyo - Readyset - RisingWave - Timeplus - Pathway - Dozer - ReadySet - Snowflake dynamic tables - Native materialized views in OLTP databases - Just having a stack of views in your db - Poor man's MVs with triggers
All subtly different on every spectrum from consistency, UDF support, operator support, latency, scaling/state limits, source/sink integrations, and compatibility with existing protocols.
What seems unique is the focus on "writebacks to the source without Kafka/Connect in between", instead of having either a built-in cache, serving as a stream processor, or both. It looks like the built-in cache is still available through the FDW deployment pattern.
They note that relative to the source tables they are eventually consistent (of course, unless you want to delay transaction writes) but it's not clear what other consistency aspects they respect (such as preserving transactions end to end).
Overall this looks like it's designed to overcome materialized view limitations (which in popular OLTP dbs are pretty severe w.r.t. either what operations are supported, latency, or both) compared to other solutions that basically move the action downstream...curious if it will see much use, or if they'll inevitably introduce sinks and direct access to see if they can compete in the "live ODS" segment with Materialize and RisingWave.
edit: to make my comment more clear: this is a new entrant in a crowded space with several sophisticated, established players and the main differentiation is the deployment pattern. I'd be curious to know if anything else sets them apart