The 120 MiB/s encode ceiling is the cost of the mode competition. that's where the ratio comes from. At 800-1600 MiB/s off a digitizer, fc is the bottleneck no matter what transport sits behind it; for that regime zstd-3 or lz4 are the better fit, or fc further down the pipeline on aggregated/decimated data.
You're also right on int/short. fc's modes look for IEEE-754 bit patterns, so doubles that started life as rescaled ints lose the structure those modes exploit. A native int16/int32 path is on the list.
For the wiring itself: I have a sister single-header library, vibe (https://github.com/xtellect/vibe), built for this exact pattern: length-prefixed TCP/IPC framing on Linux, with a `telemetry_sink` example close to the edge-sensor --> cloud-ingest case. Producer compresses with fc, ships framed bytes through vibe, consumer decompresses. Doesn't solve the throughput ceiling, but handles the producer/consumer setup cleanly.
edit: i think the comments is flagged automatically because I used `vibe` (bad name I know) :)
I need to add it to the benchmark. My expectation is that OpenZL should be strong when the enclosing format is known and SDDL can separate typed fields cleanly. Running both on the same f64 arrays will give some information
Yes it is. The mismatch is mainly representation and purpose: audio is usually int16/int24/float32 PCM, and audio codecs often exploit perceptual loss. fc is lossless and currently tuned for float64 streams.
I have an XOR128-style mode and a byte-transpose/byte-split-like mode, but I should not claim that as a proper Chimp128 or Arrow Parquet byte-stream-split comparison yet. I willadd direct baselines for Chimp128 and Arrow/Parquet BSS+zstd to the harness.
I’m regarding that term loosely here- in this case it is 'try several representations/codecs for a block and store the winner.' Similar ideas show up in columnar formats choosing encodings per column/page, OpenZL selectors (asother commenters pointed here), and shuffle/transpose + backend-compressor pipelines. fc’s version is much narrower: a tournament among f64-specific modes per block.
That’s a fair description. One mode does not dominate in my current harness; the winning mode varies quite a bit by dataset/block. If real workloads show one or two modes dominate, I’d rather simplify the portfolio :) For now the extra encode CPU is intentional: spend time once, get smaller blocks and fast parallel decode.
Thanks, this looks super relevant. I think the transferable part is the per-block selectrover predictors, strides, deltas, exponent/mantissa-ish structure, byte transpose, fallback raw/LZ, etc.sddl2 looks like a natural place to try some of that.
It is intended t obe mainly source agnostic (will try to add custom source predictors too). The idea is to treat input as an ordered stream of doubles and look for numeric structure like repeats, smooth deltas, fixed increments, or low-entropy bits. Target presentlyis scientific/time-series/simulation/analytics data, not photos or sound.
I built "fc", a C library for compressing streams of 64-bit floating-point values without quantization.
It is not trying to replace zstd or lz4. The idea is narrower: take blocks of doubles, try a set of float-specific predictors/transforms/coders, and emit whichever representation is smallest for that block.
It is aimed at time-series, scientific, simulation, and analytics data where the numbers often have structure: smooth curves, repeated values, fixed increments, periodic signals, predictable deltas, or low-entropy mantissas.
The API is intentionally small: "fc_enc", "fc_dec", a config struct, and a few counters to inspect which modes won. Decode is parallel and meant to be fast; encode spends more CPU searching for a better representation.
Current caveats: x86-64 only for now, tuned for IEEE-754 doubles, research-grade rather than production-hardened.
I think this is interesting too sqlite a as the coordination boundary: business state, queue state, stream offsets, retries, and acks all sharing one transactional substrate. The 1ms polling is getting a lot of weight in the thread though :)
In Cactus the fast path is local. A worker pushes its own continuation onto its own deque, runs the child, and later tries to reclaim that continuation locally. Other workers only touch that deque when they become idle and steal.
With one global deque, every fork/pop/steal hits the same shared structure, making it a cache-coherency hotspot.
Per-worker deques make the common case mostly uncontended; stealing is only the load-balancing fallback.
So a global deque is simpler, but it scales worse.
I ran into this rabbithole while writing an x86-64 asm rewriter.
xor was the default zeroing idiom.I onkly did sub reg,reg when I actually want its flags result. Otherwise the main rule is: do not touch either form unless flags liveness makes the rewrite obviously safe. Had about 40 such idioms for the passes.
I feel like LLMs are just forcing me to realize what writing actually is. For me, writing is basically a mental cache clear. I write things down so I can process them fully and then safely forget them.
If I let an LLM generate the text, that cognitive resolution never happens. I can't offload a thought i haven't actually formed - hence am troubld to safely forget about it.
Using AI for that is like hiring someone to lift weights for you and expecting to get stronger (I remember Slavoj Žižek equating it to a mechanical lovemaking in his recent talk somewhere).
The real trap isn't that we/writers willbe replaced; it's that we'll read the eloquent output of a model and quietly trick ourselves into believing we possess the deep comprehension it just spit out.
It reminds me of the shift from painting to photography. We thought the point of painting was to perfectly replicate reality, right up until the camera automated it. That stripped away the illusion and revealed what the art was actually for.
If the goal is just to pump out boilerplate, sure, let AIdo it. But if the goal is to figure out what I actually think, I still have to do the tedious, frustrating work of writing it out myself .