We built our data transformation engine with the Wasm runtime(redpanda.com)

34 pointsby sorenbsvor 4 Jahren21 comments

We built our data transformation engine with the Wasm runtime

https://redpanda.com/blog/data-transformation-engine-with-wasm-runtime/

22 comments

dangvor 4 Jahren

This is a good post but besides the non-organic upvotes, it looks like your friends/fans/teammates have seeded the thread with comments and questions. This is a super bad idea. HN readers can smell this kind of thing in the parts per million and will flame you for it. It's also against the rules—this is in both the guidelines and the FAQ:

https://news.ycombinator.com/newsguidelines.html

https://news.ycombinator.com/newsfaq.html

agallegovor 4 Jahren

i got a notification in slack. there is less organization than you think :D. if we were, we'd have done this w/ the last 15 posts.

dangvor 4 Jahren

Slack messages are how most of this works. It doesn't change the point.

agallegovor 4 Jahren

dunno what to say. the answer doesn't change.

abd12vor 4 Jahren

Nice post!

Question for you -- what are the performance implications of "re-keying" my records during the transform? Should I try to keep the same key for my transformed record as from my original record so that they align with the same partitions, or are they likely going to get sent to different partitions across different brokers anyway?

followtherhythmvor 4 Jahren

Good question! You can't re-key your records but this done is by design. We want the materialized partitions to reside on the same brokers as their sources so there is no network overhead for performing the transform, and it occurs on all nodes. So by choosing this design we traded off more CPU for lower latency and less network bandwidth.

markpapadakisvor 4 Jahren

We have been running hundreds of services that effectively consume from various topics, perform mostly simple transformations and produce back to other topics. Providing this kind of functionality makes a lot of sense.

We are using TANK, and that functionality is not available there, but given the benefits such a feature provides, we will likely use RP as well in order to save on development time and number of distinct services we use for various events streams transformations.

hammersmithvor 4 Jahren

Hiya! Excited to try this, The devil is of-course in the details :) What are the throughput implications for the rest of the cluster? IS there any way to profile the transforms usage as a percentage of the whole? How much of the API is exposed? can it create topics?

followtherhythmvor 4 Jahren

Hi, of course there are :) So the transformations are deployed in a separate NodeJs process per node. You can use familiar tools to take a look at its cpu + memory usage.

As for how it affects redpanda, currently we are working on exposing metrics via Prometheus. All of this work occurs on background threads, so the only way it would affect throughput is if enough coprocessors are deployed with enough actual work that would eventually start to fill up our seastar reactors.

pandemiclattevor 4 Jahren

can materialized topics have custom properties? what if I want a longer retention in a materialized topic, perhaps I'm just counting clicks and I don't want the whole header data from that topic.

followtherhythmvor 4 Jahren

Currently they cannot, as they share all of the same properties as their respective source topics, but this idea is good! I'm adding it to our list of design enhancements.

pandemiclattevor 4 Jahren

That's awesome! I would love to try that option one day! Thanks!

northstar702vor 4 Jahren

Are there other wasm runtimes to consider? Why v8?

followtherhythmvor 4 Jahren

Yup, as of now we are leveraging Nodejs for the technical preview. We also have another preview using embedded v8 for inline transforms, that is your transform runs in-process and apply() boils down to a function call. We are still evaluating other runtimes like wasmer as well, v8 was initially chosen for its maturity and community support.

PeterCorlessvor 4 Jahren

Great minds think alike! We're adding Wasm support to ScyllaDB, too.

https://github.com/scylladb/scylla/pull/9108

Folks can hear a talk on the implementation at Scylla Summit next week [free to register and yes, will be available free on demand after].

We'll also have a talk by Redpanda's Denis Rystsov.

https://www.scylladb.com/summit/

agallegovor 4 Jahren

I'm pretty excited to see the wasm chat too. very cool. I think that wasm is going to do to server side what javascript did for the web in the 90's

[deleted]vor 4 Jahren

(1)

followtherhythmvor 4 Jahren

Hi I'm one of the authors of the blog post, happy to answer any questions here !