HackerTrans
TopNewTrendsCommentsPastAskShowJobs

danielgafni

no profile record

Submissions

Show HN: Metaxy – versioning for multimodal data pipelines

docs.metaxy.io
3 points·by danielgafni·قبل 5 أشهر·1 comments

comments

danielgafni
·قبل 4 أشهر·discuss
Hey, I’m the author.

At the bare minimum tests of unchanged code and dependencies would be skipped (cached) as well in this setup.

More sophisticated rules would have to be set up by hand, but again it’s easy to do with Dagger (as you can expected any kind of logic there).

But the whole point of using Dagger in this setup is to get tests caching out of the box, and for that you need to assemble the container correctly (by only including relevant dependency files).
danielgafni
·قبل 5 أشهر·discuss
Hi HN, I'm Daniel, an ML Ops engineer.

I work at Anam and am responsible for the preparation of our training data.

I built Metaxy to avoid re-running expensive data processing steps on irrelevant upstream changes.

It's an open-source Python framework that can track granular sample versions and is infrastructure-agnostic.

See the full blog post here: https://anam.ai/blog/metaxy.

Happy to answer questions.

P.S. While Claude Code helped me a lot, this is *not* a vibe-coded project: I treated AI code like any line could be wrong, created a massive test suite, and Metaxy has been running in our production for about 2.5 months now.