Until now, container was only way to provide isolation boundary which as process. With WASM, we can provide very fine level isolation and execution control.
There is a significant difference between doing stream processing at edges (client, producers) vs. inline. Similar to performing SQL query in client vs. done by the database server. Think of the WASM inline module as a stored procedure but for the stream (omitting other implementation details).
> How does it prevent that if a stream being processed contains > PII data?
At the execution time, the WASM module can't access any memory outside it is given. Fluvio's SPU control data flow from the stream to the WASM module. For the PII use case, data is only decrypted just before sent over to the WASM engine. And since the WASM module can't access any data or have access to any host-based resources, it can't leak anything.
The key architecture advantage of in-line processing is that we can move computation as close to storage as possible reducing cost and latency
It's a streaming-based data platform which means "data stream" is a fundamental first-class citizen similar to how the table is on the relational database. The key benefit is that data can be move without doing polling (or writing job scheduler) with is how done in the traditional batch-based ETL. Compared with the legacy streaming platforms, Fluvio fits better in a cloud-native environment, with less resource required in terms of memory and CPU load usages which means it is significantly cheaper. With our WASM inline programmable engine, you can perform data computation near storage, eliminate unnecessary data movement, reduce network transmission costs and delays, and eliminate security risks. We will start to publish our performance and other stats. Our SPU (Streaming Processing Unit) runs in 1/10 of memory as a Kafka broker (which requires a min of 1G memory) on a simple benchmark.
Thanks. A key reason for starting Fluvio is to democratize data. Make data streaming & data engineering accessible to everyone and remove barriers much as possible.
You can compile almost any language to WASM not just Rust. For example, Python, Go, Javascript: https://github.com/appcypher/awesome-wasm-langs.