My biggest gripe is that he's comparing probabilistic models (LLMs) by a single sample.
You wouldn't compare different random number generators by taking one sample from each and then concluding that generator 5 generates the highest numbers...
Would be nicer to run the comparison with 10 images (or more) for each LLM and then average.
Btw, regarding a bit older pre-Flakes commits, at least some packages will error out while nix-installing on M1 (or newer) macbooks because at that point nix packages didn't anticipate macbooks ever ditching x86 for aarch... Hope this info helps you out, at least took me some time to realize that the random obscure error was caused by this. Can be almost always fixed by appending "--system darwin-x86_64" to the nix command to use Apple's Rosetta binary translation, cheers.
Thanks for the answer! Looking forward to the followup post :D
And yeah, nix naming conventions are sometimes a mess. Your approach does seem much nicer.
Last question if you have a bit more time. How come that you don't have every version of a package available? Is it because you only parse the Hydra stable 23 builds (which I guess aren't updated on every version) or because you parse them only periodically like Lazamar's Nix Package Versions?
> yet if I understand correctly, it often has one version per package per Nix version?
Both yeah and no. Taking the example of nodejs, you have all the supported major releases like nodejs_16, nodejs_18, etc. for each Nix revision[1], but within each revision you only have a specific version of the package. E.g. nodejs_16 is 16.3.1 for revision at date X. There are benefits and drawbacks of managing a package repo like that, with revisions which implicitely pin all packages and it's dependencies at once, but it's too long of a topic to write about here.
> For example the curl package [1] has tons of versions [...]
So does it in nix (curlMinimal, curlWithGnuTls, curlHTTP3, etc)[2]. In that regard nix and Spack are similar.
[1]: A revision is like a shapshot of all the packages at a given point in time.
First off, amazing tool and thanks for doing the nix community a favor!
If you could expand more on the architecture of the tool I'd be very curious! Like what goes all on from parsing Hydra builds to getting a useful package versions DB and what were the challenges while building it? Also, how are you merging all the nodejs packages (nodejs_20, nodejs_18, ...) into one “nodejs”?
I'm also in the process of building something very similar (didn't know the idea was that popular haha), but I'm not parsing Hydra builds but the output from `nix-env -qa --json` to get all the package versions and it's proving more involved than I anticipated (:
https://help.ente.io/self-hosting/