Hey! We have just published a review and benchmark of different uncertainty estimation techniques [1], it might be interesting to you if you want to get a general understanding of works and what doesn't in the specific case of LMs.
You are right that uncertainty is a kinda loosely defined term. Usually people mean that it's a kind of proxy to the probability that the output of the model is correct in some sense.
It's also true that uncertainty can be decomposed into "flavours". The simplest and most discussed decomposition is into aleatoric and epistemic kinds of uncertainty. Epistemic uncertainty (or model-based uncertainty) usually refers to the case, when poor output is a result of the model being presented with the kind of input which it never saw before, and should not be expected to handle correctly. Aleatoric uncertainty on the other hand is thought to be intrinsic to the data itself, think of the natural ambiguity of the task, or noisy labelling.
People in the field of uncertainty estimation are very much concerned with developing methods of quantifying these different types of uncertainty, and different methods can be more sensitive to one or the other.
When people in this field compare various methods of quantifying model uncertainty, they often perform what is called rejection verification. Basically, you continuously reject data points where uncertainty is high, and see how average quality of the remaining outputs increases. A good uncertainty estimate is highly correlated with output quality, and thus low-uncertainty outputs should have higher average quality.
We use exactly this approach in our recent benchmark of uncertainty estimation approaches for LLMS [1] and have an open-source library under development [2] which allows for such benchmarking. It also can produce uncertainty scores for a given model output, so ppl in industry can integrate it into their applications as well.
The idea behind semantic entropy (estimating entropy of distribution over semantic units, instead of individual sequences in the output space) is great, but it's somewhat naive in the sense that it considers these semantic units to be well-defined partitions of output space. There is further generalization of this approach [1] which performs soft clustering of sampled outputs based on a similar notion of semantic equivalence between them.
But even with this in mind, there are caveats. We have recently published [2] a comprehensive benchmark of SOTA approaches to estimating uncertainty of LLMs, and have reported that while in many cases these semantic-aware methods do perform very well, in other tasks simple baselines, like average entropy of token distributions, performs on par or better than complex techniques.
We have also developed an open-source python library [3] (which is still in early development) that offers implementations of all modern UE techniques applicable to LLMs, and allows easy benchmarking of uncertainty estimation methods as well as estimating output uncertainty for deployed models in production.
Yep, he has an extremely charming personality judging from his videos. Something similar is about Robert Sapolski, also from Stanford. Love watching these guys lecture. Natural born teachers.
A somewhat relevant question: what is a good way to get into longevity research/research support for a software engineer with a degree in optical engineering, who recently started refreshing college math?
Man I wish I knew about this when I started my last pet project. Just finished the first major milestone, huge part of which was to create a basic bootstrap interface. This would've saved me a lot of time and effort, I'm really bad at markup and UI/UX.
How useful do you think is studying general statistics from, for example, OpenStat vs. directly learning from ML-related courses like the one you mentioned?
[1] https://arxiv.org/abs/2406.15627