Am I wrong to be somewhat peeved by the use of "RAG" in these contexts? I always read things like this, and wonder if instead the author should be saying "Semantic Retrieval" or something something Vector, etc. Retrieval augmented generation captures tool-use, and; semantic search of course is really just a tool under the hood.
To make an anology, in my mind, this is akin to saying "fuel air mixture system" when referring to direct fuel injection specifically, when of course, a carburetor also lives in that category.
Reading your comment reminded me; I actually did something quite similar at $MY_BETTER_STARTUP! My approach is slightly different, however, employing what I like to call State-Horizon-Aware-Rercursive-Threaded-Graph-Position-Topology.
With a 400% increase in words, $MY_WAY_BETTER_HARNESS looks to be about four times as performant. SHARTGPT isn’t just a harness engineer's playground — it's a the cyber jungle gym that frees them from $MY_HARNESS.
If you want, I can even include a sentence or two that will really tell those potential investors why they should shower you with money instead of the other commenter! Just say the word!
(Looking for lead investors for our angel syndicate btw! DM me if interested)
Sounds like user error to me. Codex gives an llm a tool to allow it to use shell in the context of the host and user in which it is running. If a resource is sensitive, and accessible in that context, then the user is doing something wrong. Would you change your practices if you treated your coding agent as an untrusted human ssh'd under the identity you use for it?
In any case. There are solutions in the comments on the issue, as well as this hn thread.
Ah, I wasn't aware things regressed there. Yea certainly workarounds n soft fork sorts of things definitely would work- but thats a bummer than things have changed.
From watching Pr's and issues- seems like openai at least wants to come across as if theyre supporting non-oai models :/
I keep butting into the question of; why opencode, when you've got codex available? Codex is open source as well, and i can't seem to picture a situation where one would want Opencode over Codex.
As far as I can tell, they tick the same boxes- but one has the support of a big boy model provider.
The "Very good" I'm referring to is far better than only 99%. I can't offer solid stats off the top sadly, so you'll have to just take my word for it ;)
I'll take the opportunity to note that if you're running solid evals, you'll have data to back the efficacy of your system. If you are seeing a hallucination rate of 1%, then you certainly should be working on your harness/toolset/context/prompting etc.
Saying "1% hallucination rate..." is akin to saying "30,000mi lifespan for [modern japanese make engine]". Something is wrong.
I find papers/articles which discuss solutions that rely heavily on a model in the middle unreadable, if the models used are not discussed.
The data you need to get into context for a small model, vs a big boy frontier model, vs a fine tuned open weight big boy- are all very different. I can understand what they're doing here, and most of the 'why', but- not all of the why.
I have gone through this process and evaluated the results. Maybe you're referring to their comment as written, but going through what OC described + handholding leads to very good results in my experience.
Nvidia Openshell solves most of the hard problems I've run into while building stuff in this space.
Observability is, for my purposes, solved by a given framework supporting OpenTelemetry.
Guardrails is where I've gotten the most value of openshell being a neat package. Agent workload scope is written as policy in openshell, and capability is backed by openshell handling all execution.
Monitoring/deployment/versioning is helped as well, depending on how agents/runners are slotted into the system. Deployment namely is quite well supported- openshell has kube/helm bits that are experimental atm, but seem like a logical approach imho.
Evals and a/b testing isnt something ive explored in depth, considering that agents with composable tool sets + frontier models are beyond my expectations already.