On the contrary, I agree - while there is certainly hype being generated around AI, particularly generated by the "VC hype cycle", the fundamental advancements we've made with LLMs are quite real.
Part of the reason I wrote this is to separate the signal from the noise and why one should be {cautiously, more tempered} optimistic in the medium term.
pre-training is developing the language model's base understanding of conditional word probabilities.
SFT and RLHF is attempting to further guide the model in terms of steerability + alignment of output.
In fact, the InstructGPT authors were worried about losing the pre-trained model's underlying probability distribution, so they try a version where it penalizes the model deviating too significantly from the original distribution (using KL). I don't remember them seeing a significant difference in performance.