I am afraid that this post is missing the biggest point.
Given a prompt (or task) how do I evaluate if it is a "simple" task that should be executed by a small model or if it is a complex one that may need a SOTA model?
Why I believe to be different this time is that the engineer themselves feel this need AND this is a real need making software delivery actually faster.
> Give an agent the right interfaces and it becomes less conversational and more ambient. It no longer needs to constantly ask, explain, summarize, and negotiate. It can stay in the background, react to changes, and make steady progress with less supervision and less noise. That is closer to Weiser’s vision: calm technology, but for machines.
I tend to agree quite a bit.
I created a ambient background agent for my projects that does just that.
It is there, in the background, constantly analysing my code and opening PRs to make it better.
The hard part is finding a definition of "better" and for now it is whatever makes the longer and type checker happy.
Homework and home assignments are not really a way to grade students. It is mostly a way to force them to go through the materials by themselves and check their own understanding. If they do the exercises twice all the better.
(Also nowadays homework are almost all perfect scores)
Which is why LLM are so deleterious to students. They are basically robbing them of the thing that actually has value for them. Recalling information, re-elaborating those information, and apply new mental models.
My SO is a college educator facing the same issues - basically correcting ChatGPT essays and homework. Which is, beside, pointless also slow and expensive.
We put together some tooling to avoid the problem altogether - basically making the homework/assignment BEING the ChatGPT conversation.
In this way the teacher can simply "correct"/"verify" what mental model the student used to reach to a conclusion/solution.
With a grading that goes from zero point for "It basically copied the problem to another LLM, got a response, and copied back in our chat" to full points for "the student tried different routes - re-elaborate concepts, asked clarifying question, and finally expressed the correct mental model around the problem.
I would love to chat with more educators and see how this can be expanded and tested.
For moderately small classes I am happy to shoulder the pricing of the API.