What a great read!
You got me at the base64 oddity. I also stumbled over this, while trying to dodge some LLM limitation. (was trying to generate images in a time before multimodal was a thing. it only worked to a degree).
Exactly my point of view. For the most part I do not root for my preferred technology, but rather try to inform my powers about the caveats I see. This way at least the right aspects to check have a chance to enter the debates above my payroll.
I second that thought. There is a pretty well cited paper from the late eighties called "Multilayer Feedforward Networks are Universal Approximators". It shows that a feedforward network with a single hidden layer containing a finite number of neurons can approximate any continuous function. For non continous function additional layers are needed.