I don't disagree, but I'm wondering if there's any evidence of this available.
> After all, LLMs have more natural language training data than JSON training data.
While that is true, data also doesn't usually look like natural language (i.e. a collection of financial records). And when it does (i.e. a collection of chat messages), I wonder if it's more confusing if it's unstructured, even if small.
I expect most frontier models to handle these cases just fine either way, so it may largely depend on context—specifically, how much there is, and where the attention shakes out. Ultimately, a claim one way or the other, for something this context-dependent, would have to be backed up by a lot of testing and would probably conclude that, "in most cases, you should do this"
And whether or not it actually is feasible, many are betting millions that it is, and marketing it as possible to keep the innovation machine running until we achieve it
> ... I always say about [a language model], the [linguistic] appearance makes a promise about what it can do. [Clippy] was this little [cartoon paper clip]. It didn’t promise much—you saw it and thought, that’s not going to [write the next great novel]. But you can imagine it [offering limited help]. But [human language] sort of promises it can [write] anything a human can. And that’s why it’s so attractive to people—it’s selling a promise that is amazing.
> After all, LLMs have more natural language training data than JSON training data.
While that is true, data also doesn't usually look like natural language (i.e. a collection of financial records). And when it does (i.e. a collection of chat messages), I wonder if it's more confusing if it's unstructured, even if small.
I expect most frontier models to handle these cases just fine either way, so it may largely depend on context—specifically, how much there is, and where the attention shakes out. Ultimately, a claim one way or the other, for something this context-dependent, would have to be backed up by a lot of testing and would probably conclude that, "in most cases, you should do this"