Calibration (in a binary context) basically means that the confidence of a model/score matches the probability that a particular label is positive or not.
For instance, a calibrated classifier for a coin flip predictor should output 50-50. A poorly calibrated classifier would output higher confidence for heads/tails.
Good question! We do know from OpenAI's system card from GPT-4 that the post-trained RLHF model is significantly less calibrated compared to the pre-trained model, so it's a matter of speculation that something similar is occurring. However, it's more of a hunch more than anything. I would be curious if it's possible to reproduce this behavior, or the impact of distillation on calibration.
In fact, for many models you can remove refusals rather trivially with linear steering vectors through SAEs.
https://www.alignmentforum.org/posts/jGuXSZgv6qfdhMCuJ/refus...
Additionally, you can often jailbreak these models by fine-tuning the model on a handful of curated samples.