yung_lean · HackerTrans

21 gün öncediscuss

This isn't using WASM to solve the "how can I make my file format compatible with more programming languages?" problem. This is trying to solve the "how can I add new encodings to my file format without making everyone update their code?" problem. The former would rightly be solved with C bindings that anyone can link with if they want. The latter might not seem like a big deal, but it's been the main blocker advancing the parquet format. Most people end up not caring about new advanced encodings and just write parquet files with the most compatible feature set.

yung_lean

21 gün öncediscuss

A big problem with parquet, which this aims to replace, is that it's hard to add new encodings because everyone wants to stay compatible with old readers. Embedding the decoders in the file as WASM solves this problem since in theory, old readers will be able to read new files by just using the provided WASM to decode a column whose format the reader doesn't recognize.

So this is really about making a file that is forwards compatible in a way that lets you push the standards more than existing formats.

yung_lean

21 gün öncediscuss

Yeah this was a research project, it doesn't look like this is getting any adoption

yung_lean

geçen yıldiscuss

In my experience, understanding the rules of the borrow checker is not enough to be able to write rust code in practice. For example, ~6 months into using rust I was stumped trying to move data out of a mutable reference. Trying to do this directly by dereferencing gives compiler errors like "cannot move out of `*the_ref` which is behind a mutable reference". If you know rust, you're probably either yelling "you idiot! you can't move out of mutable references!" or "you idiot! just use std::mem::take!" (the latter of course being the right way to do this) but that's not obvious from the borrow checker rules.

My experience learning rust has been like a death by 1000 cuts, where there's so many small, simple problems that you just have to run into in the wild in order to understand. There's no simple set of rules that can prepare you for all of these situations.

comments