nsingh2 · HackerLangs

vor 4 Tagendiscuss

My own experience: much less context bloat, and hidden system prompts. Can tune it to your own workflow, instead of dealing with whatever hidden system prompt Claude Code forces on you.

nsingh2

vor 4 Tagendiscuss

To be fair, Pi only had to be useful. Claude Code had to solve the difficult unsolved problem of making a terminal print text, so naturally they built a small game engine [1]

[1] https://x.com/trq212/status/2014051501786931427

nsingh2

vor 5 Tagendiscuss

I think it might affect real work if part of it requires a lot of thinking, i.e. something similar in nature to a puzzle.

There seems to be something wrong with the "commentary" channel related intermediate updates, maybe the model gets confused about what's an intermediate update vs what's the final answer? [1]

[1] https://github.com/openai/codex/issues/30364#issuecomment-48...

nsingh2

vor 6 Tagendiscuss

This is preliminary, but it seems like it might somehow be related to the `## Intermediary updates` system prompt that's provided to the model. Seems like it forces the model to stop thinking and return early to provide updates. Removing that entirely makes all runs succeed [1].

I wonder if it's somehow getting confused between what's supposed to be an intermediate update vs the final result.

[1] https://github.com/openai/codex/issues/30364#issuecomment-48...

nsingh2

vor 6 Tagendiscuss

nsingh2

vor 8 Tagendiscuss

Codex apparently added this too, I only noticed a few days ago: https://github.com/openai/codex/issues/28969

I'm usually at my desk, and get a notification when it stops to ask a question, so I've never accidentally had it timeout yet. Still an annoying change.

nsingh2

vor 9 Tagendiscuss

A highly detailed specification is not what I mean here. It's closer to plugging in a few sentence descriptions (or a totally cluttered brain dump) and having the model interview you to help pin down critical details before continuing.

In my own work, it's usually been a few critical assumptions the model made silently (and I never even though of initially) that end up being the difference between passable results the first try, and me having to go back and fix things. Occasionally some questions force me to rethink the problem entirely.

I basically always begin any long-running session with this kind of brainstorming. I don't find the existing plan modes in Claude Code/Codex to be critical enough.

nsingh2

vor 9 Tagendiscuss

Refusing to sufficiently specify a task and hoping the model guesses correctly is not being productive. Again, these models still don't really ask questions when they should. You have to explicitly tell them to.

Specifying the problem is not extra work separate from solving it. If you skip that step, the ambiguity gets pushed into the model’s assumptions. Then you get a plausible looking answer to the wrong problem and have to waste time backing out of it.

LLMs are not magic machines that can read your mind.

nsingh2

vor 9 Tagendiscuss

Why supply underspecified requirements in the first place? Both models are good at challenging assumptions/edge cases and asking questions to clarify, but seemingly only when explicitly asked (i.e. something like a "brainstorm" skill).

I don't think either harnesses do enough to encourage the model to challenge all assumptions and ask questions, maybe because users might find it annoying. That step is basically a requirement IMO.

I've found all of the GPT-5 models to be very nit-picky, useful for code review and mathematics (important for my work), but seemingly gets in the way of "aesthetic" code, e.g. overly defensive code to cover all edge cases, even if unlikely.

There is seemingly also a tradeoff between flexibility vs instruction following. In my experience Opus will sometimes ignore instructions but can "fill in the blanks" more, vs GPT-5.5 follows instructions better but perhaps at the cost of rigidity.

nsingh2

vor 10 Tagendiscuss

Cost per task is shockingly high. More expensive than Opus 4.8, second in place to Fable.

Cost per task data is only available for max effort though, might just be very inefficient at that effort level.

nsingh2

vor 10 Tagendiscuss

That seems to only be true for the "Agentic Search" benchmark. That benchmark in particular is a bit weird, because Sonnet 4.6 effort levels had a relatively small effect, so Sonnet 5 med is basically comparable to all effort levels of Sonnet 4.6.

nsingh2

vor 12 Tagendiscuss

> Don't use it to write code

One of the things I've started doing is to write the high level scaffold myself, create stubs (with comments) and have it fill out the details. I haven't done this for long, but I feel like my mental model of the code rots less, and I can review code faster.

I used to do this in the past, when these models were not all that capable, now there is a huge temptation to leave everything to the model. But long term I lose track of whats going on, and have a hard time even communicating with the model about the changes I want. Just a total decay of my mental model of the system, and it's really hard to recover from that state.

nsingh2

vor 12 Tagendiscuss

This is false. A process needs read permission on the relevant `/dev/input/` device, typically by running as root or as a user in a group like `input`. Normal desktop users generally should not be in the input group. Regular applications receive keyboard input through the compositor/windowing system.

nsingh2

vor 14 Tagendiscuss

Yea these plots are too noisy and dense. Especially that second one, lines all over the place.

nsingh2

vor 14 Tagendiscuss

From what my own experiences are, and what's on their checkout page, $100 is 5x base usage and $200 is 20x. If $100 was 10x, then I personally would drop down. They want people to go to the highest tier.

nsingh2

vor 14 Tagendiscuss

I'm really getting sick of reading about safeguards and what I'm not allowed to do on every model release.

nsingh2

vor 18 Tagendiscuss

To follow up on this, I had it solve a nasty ODE problem that I saw in the recent Mathematica 15 release post:

    Solve the following first-order ODE for f(x):

    ((-1 - 2*x)*f(x)*tan(1 + x - exp(-61 - 2*x)*f(x)/x)
    + exp(61 + 2*x)*x*(1 - x*tan(1 + x - exp(-61 - 2*x)*f(x)/x))
    + x*tan(1 + x - exp(-61 - 2*x)*f(x)/x)*f'(x)) = 0

    Find the general solution f(x).

And surprisingly it found a valid solution! Extra impressive because it runs 25 tok/s on my measly RTX 2070 super.

    f(x) = x*exp(61 + 2*x)*(1 + x - arccos(C/x))

    C is an arbitrary constant.

Apparently Mathematica 14.3 couldn't solve this ODE.

nsingh2

vor 18 Tagendiscuss

Lots of confusion about what this model is actually focused on.

It is a cheap specialist for closed-world, verifiable reasoning tasks like math, self-contained coding problems, and similar.

"Closed-world" means the needed information is already in the context. It is not a tool-using agent that can discover missing context. "Verifiable" means answers are hard to generate but easy to check.

So no open ended research, repo wide agent work, factual Q&A, or SVG generation. More of a compact reasoning module for bounded problems.

nsingh2

vor 18 Tagendiscuss

The lack of tool use will hinder it a lot I think, since bug hunting requires collecting context across a code base and stitching it together. It might be good in a more narrow sense, i.e "is there a bug in this block of code" and not considering how it interacts with the rest of the code base.

That's also more aligned to its leetcode style training data, the code under test is fully in the context window. It might be interesting to have a bigger tool use model go through the effort of collecting the context, and feeding it into this kind of model for analysis only. It becomes more of a thinking tool, instead of the orchestrator.

nsingh2

vor 18 Tagendiscuss

This model doesn't support tool calling, was not part of its training. It's focused on Python (and I think C++) competitive programming and mathematics tasks, i.e. tasks with verifiable rewards. So if you have a task that fits that description, the size-to-capability ratio is good.

These kinds of models might be more useful as tools to be used by larger orchestrator models, than being the orchestrators themselves.