Good points. But language models are usually then fine-tuned for specific tasks. Contextual reasoning in NLG is a specific area of research that attacks some of the points raised and certainly BERT wasn't pretrained to solve all tasks in NLP/ NLG area of research.