Congratulations on the strong reception of min-p. Very clever!
We may be talking about two orthogonal things here. And also to be clear, I don't care about theoretical guarantees either.
Now, min-p is solving for the inadequacies of standard sampling techniques. It is almost like a clever adaptive search which other sampling methods fail at (despite truncations like top-k/top-p).
However, one thing that I noticed in the min-p results was that lower temperatures were almost always better in the final performance (and quite expectedly the inverse for creating writing). This observation makes me think that the underlying model is generally fairly good at ranking the best tokens. What sampling allows us is a margin-for-error in cases where the model ranked a relevant next token not at the top, but slightly lower.
Therefore, my takeaway from min-p is that it solves for deficiencies of current samplers but its success is not in contradiction to the fact that logprobs are bad proxies for semantics. Sampling is the simplest form of search, and I agree with you that better sampling methods are a solid ingredient to extract information from logprobs.
That has been my understanding too. More generally, a verifier at the end certainly helps.
In our paper [1], we find that asking a follow up question like "Is the answer correct?" and taking the normalized probability of "Yes" or "No" token (or more generally any such token trained for) seems to be best bet so far to get well-calibrated probabilities out of the model.
In general, the log-probability of tokens is not a good indicator of anything other than satisfying the pre-training loss function of predicting the "next token." (it likely is very well-calibrated on that task though) Semantics of language are a much less tamable object, especially when we don't quite have a good way to estimate a normalizing constant because every answer can be paraphrased in many ways and still be correct. The volume of correct answers in the generation space of language model is just too small.
There is work that shows one way to approximate the normalizing constant via SMC [2], but I believe we are more likely to benefit from having a verifier at train-time than any other approach.
And there are stop-gap solutions to make log probabilities more reliable by only computing them on "relevant" tokens, e.g. only final numerical answer tokens for a math problem [3]. But this approach kind of side-steps the problem of actually trying to find relevant tokens. Perhaps something more in the spirit of System 2 attention which selects meaningful tokens for the generated output would be more promising [4].
I use Astro + Cloudflare Pages for my website [1]. I document the key bits of my stack here [2] for completeness.
I've been very happy with Astro because it is a good example of low floor and high ceiling software. I can start with plain HTML, make it more flexible with Astro language (still very close to HTML), make authoring easier with Markdown (+ lifestyle extensions from Remark/Rehype), and extend to frameworks like React on a need basis (which I use for some pages where I use maps).
The best thing that one can do for themselves to develop the creative "muscle" is to _own_ their time.
Unfortunately, I am yet to feel even close to such a breakthrough. I think very few are fortunate to afford such kind of luxury (as the author alludes to as well). There is always something to deliver for, a deadline to meet (although many would argue deadlines are a forcing constraint); a life waiting to happen. With a tiny bit of envy, I feel very happy and inspired when someone does achieve the "flow" state.
On the subject of "tools" to spur creativity, I have always been skeptical. It feels similar to believing that there is a productivity app right around the corner that will unleash your potential. For me, the only true indicator of my productivity has been actually putting in the _time_, making any kind of progress along a chosen direction and then re-evaluating.
What are fellow readers here doing to _own_ their time?
We may be talking about two orthogonal things here. And also to be clear, I don't care about theoretical guarantees either.
Now, min-p is solving for the inadequacies of standard sampling techniques. It is almost like a clever adaptive search which other sampling methods fail at (despite truncations like top-k/top-p).
However, one thing that I noticed in the min-p results was that lower temperatures were almost always better in the final performance (and quite expectedly the inverse for creating writing). This observation makes me think that the underlying model is generally fairly good at ranking the best tokens. What sampling allows us is a margin-for-error in cases where the model ranked a relevant next token not at the top, but slightly lower.
Therefore, my takeaway from min-p is that it solves for deficiencies of current samplers but its success is not in contradiction to the fact that logprobs are bad proxies for semantics. Sampling is the simplest form of search, and I agree with you that better sampling methods are a solid ingredient to extract information from logprobs.