Thanks for sharing it, however I have an unrelated comment.
Maybe I am in minority here but just wanted to provide this feedback: The background animation of the blog page is really distracting and making it difficult to focus on the actual content.
I am also using conda and specifically mamba which has a really quick dependency solver.
However, sometimes repos require system level packages as well. Tried to run TRELLIS recently and gave up after 2h of tinkering around to get it to work in Windows.
Also, whenever I try to run some new repo locally, creating a new virtual environment takes a ton of disk space due to CUDA and PyTorch libraries. It adds up quickly to 100s of gigs since most projects use different versions of these libraries.
</rant> Sorry for the rant, can't help myself when it's Python package management...
I did not know that cleanrooms have classes. Apparently they used a 10000 class cleanroom which is one of the "dirtiest" grade.
Surely an astroid sample return mission is scientifically one of the most important and difficult things to accomplish, and I wonder why they did not use a higher class cleanroom for this even though they mentioned "Researchers recommend enhanced contamination control procedures for future sample-return missions to prevent microbial colonization and ensure the integrity of extraterrestrial samples."
Agreed on providing examples is definitely a useful insight vs fine-tuning.
While it is not very important for this toy case, it's good to keep in mind that each provided example in the input will increase the prediction time and cost compared to fine-tuning.
To me it boils down to what is to be measured here. With logprobs we can measure both correctness and not attempted i.e. if LLM is guessing the response.
Similar to exams where both the progress to the solution and the final outcome/value of the calculations are part of the grade.
I am puzzled why they have "asked the model" about the confidence and have not used the logprobs of the output tokens to estimate the confidence in responses.
In my use case and tests, model itself is not capable of giving a reliable confidence value where logprobs almost always provide a better view on calibration.
Maybe I am in minority here but just wanted to provide this feedback: The background animation of the blog page is really distracting and making it difficult to focus on the actual content.