For my schemas, I found LLMs really wanted to just use markdown embedded in the strings, so I've been considering doing away with the schema. I also figure that embedding markdown in a string may make it perform worse as it has to juggle nested formats, and thus escaping and such (wanted: eval for this). By replacing the json tool call with basic markdown extraction, I'd lose some structured data but gain flexibility (html would be even more flexible).
Wondering if you are referring to adherence to required data in a schema when you say LLMs do better with json vs markdown, or something else? Or perhaps to tool calls and/or strict json output being more reliably formatted for clean extraction?
That a personal website? Prod means different things in different contexts. Even then, I'd be a bit worried about prompt injection unless you control your context closely (no web access etc).
I have. In fact, I've been building my own coding agent for 2 years at this point (i.e. before claude code existed). So it's fair to say I get the point you're making and have said all the same stuff to others. But this experience has taught me that LLMs, in their current form, will always have gaps: it's in the nature of the tech. Every time a new model comes out, even the latest opus versions, while they are always better, I always eventually find their limits when pushing them hard enough and enough times to see these failure modes. Anything sufficiently out of distribution will lead to more or less nonsensical results.
That shows it knew this bit of satire more than anything. Also, the problem as stated isn't actually constrained enough to be unsolvable: https://youtu.be/B7MIJP90biM
Business problems are essentially neverending. And humans have a broader type of intelligence that LLMs lack but are needed to solve many novel problems. I wouldn't worry.
They aren't necessarily "stored" but they are part of the response content. They are referred to as reasoning or thinking blocks. The big 3 model makers all have this in their APIs, typically in an encrypted form.
Reconstruction of reasoning from scratch can happen in some legacy APIs like the OpenAI chat completions API, which doesn't support passing reasoning blocks around. They specifically recommend folks to use their newer esponses API to improve both accuracy and latency (reusing existing reasoning).
To spell it out for myself and others: approaching equivalent calculations for each individual attention block means we also approach equivalent performance for the combination of them. And with an error bar approaching floating point accuracy, the performance should be practically identical to regular attention. Elementwise errors of this magnitude can't lead to any noteworthy changes in the overall result, especially given how robust LLM networks seem to be to small deviations.
You don't get a simple request/response paradigm with claude code: 1 message from the user results in a loop that usually invokes many inner LLM requests, among other business logic, resulting in some user-visible output and a bunch of less visible stuff (filesystem changes, etc). You control an input to the outer loop: you can only do some limited stuff with hooks to control what happens within the loop. But there's a lot happening inside that loop that you have no say over.
A simple example: can you arbitrarily manipulate the historical context of a given request to the LLM? It's useful to do that sometimes. Another one: can you create a programmatic flow that tries 3 different LLM requests, then uses an LLM judge to contrast and combine into a best final answer? Sure, you could write a prompt that says do that, but that won't yield equivalent results.
These are just examples, the point is you don't get fine control.
You can't control it to the level of individual LLM requests and orchestration of those. And that is very valuable, practically required, to build a tool like this. Otherwise, you just have a wrapper over another big program and can barely do anything interesting/useful to make it actually work better.
Wondering if you are referring to adherence to required data in a schema when you say LLMs do better with json vs markdown, or something else? Or perhaps to tool calls and/or strict json output being more reliably formatted for clean extraction?