Taming randomness in ML models with hypothesis testing and marimo(blog.mozilla.ai)
blog.mozilla.ai
Taming randomness in ML models with hypothesis testing and marimo
https://blog.mozilla.ai/taming-randomness-in-ml-models-with-hypothesis-testing-and-marimo/
9 comments
Good post. I’ve been thinking about doing offline testing of LLM tasks a bit these days and have come to the conclusion that old school testing is the best until more mature features can be developed. Specifically, I mean running a power analysis to determine sample size, random sampling based on that and then running tests like a z test to see if there is a difference and between what bounds. Tests are expensive and I wish there was a better way for realizable offline evals.
i - luckily - passed my statistics exam this summer, it's however cool to visualize what's happening
qefduzh(3)