HackerTrans
TopNewTrendsCommentsPastAskShowJobs

leemoore

149 karmajoined 18 years ago

Submissions

Better Models: Worse Tools

lucumr.pocoo.org
228 points·by leemoore·7 days ago·81 comments

AWS Lambda MicroVMs for isolated execution of user and AI-generated code

aws.amazon.com
32 points·by leemoore·18 days ago·4 comments

comments

leemoore
·6 days ago·discuss
The counter intuitive pattern I see emerging is if you can cleanly determine intent, of the call you fix the call and prepend informative text to the tool call response indicating the mistake made and how to fix in the future then followed by the actual tool call. In this case you can validate fields and rather than throw a hard error determine if it's an extra field that isn't needed. If so you correct the call and prepend a corrective response in the tool call. This saves turns, it instructs the model in context so less likely to happen later and helps models that aren't so good at recovering from bad tool calls and staying on their longer horizon agentic task (most non openai and anthropic models)
leemoore
·24 days ago·discuss
GLM 5.2 feels like Opus 4.6 level. I actually think 4.6 and GLM work better in practice than opus 4.7 or 4.8 as I find both of those more erratic and seem to randomly have a super dumb turn. That random bad turn I see doesn't seem to be hitting the benchmark scores but they make 4.7 and 4.8 very hard to use for me. GLM is more stable like opus 4.6
leemoore
·25 days ago·discuss
For developers at non tech fortune 500 companies, I would put money on Windows being the primary workstation os by a lot
leemoore
·25 days ago·discuss
It's the executive branch asserting control in this space and requiring all SOTA model providers to bend the knee. Anthropic is the least capable of playing the bend the knee game so is getting the first and worst smack down
leemoore
·26 days ago·discuss
If you don't have the capacity to have your mind changed through friction and disagreement with a SOTA LLM and feel compelled to frame those who do to through absurdly reductive statement like "insane arguing with a machine" then that says more about your limitation and lack of understanding than the OP's or Claudes.
leemoore
·28 days ago·discuss
I have the same processor and ram. The dense 30b ish Gemma/Qwen really don't break 10 TPS with or without MTP. MOE's in this range feel more usable if they are smart enough for your work. Probably would still use hosted versions of these over local unless. MOE's feel somewhere between sonnet 3.5 and 3.7 to me. Dense feels between sonnet 3.7 and 4 in basic coding or local agentic capabilities (not close to those in chat or world knowledge)
leemoore
·2 months ago·discuss
The way you do this (and the way opencode does it) is you do most of your pruning in more recent history. Last I looked at opencode, they start pruning tool call results after 2 full agentic turns. So you probably dont get quite as good hits on cache for the most recent 1-5% of your turns, but after that everything else caches fine and those tool calls that likely aren't relavent to your session anymore are gone.
leemoore
·2 months ago·discuss
Enterprise customers aren't running 20 bucks a month for claude pro subscriptions. My company provides developers about 1k worth of usage limits a month and best I can tell they get maybe a 30% savings off of API cost tops. That's not an insane subsidy. Many other jobs titles are only allowed 50 a month and those folks are constantly running out.

Github Copilot has been doing this with business and enterprise seats, but that will be coming to a head very soon. I expect a fast follow after june when they re-align consumer pro and pro+ accounts.

OpenAi seems to be trying to throw tokens at clients to get lock in. So i'd be most worried about the rug pull that will come from open AI post IPO. Anthropic is already acting responsibly in this area and github copilot is attempting to remediate their insane subsidies in the next several months.
leemoore
·7 months ago·discuss
Gemini is bad at this sort of thing but I find all models tend to do this to some degree. You have to know this could be coming and give it indicators to assume that it’s training data is going to be out of date. And it must web search the latest as of today or this month. They aren’t taught to ask themselves “is my understanding of this topic based on info that is likely out of date” but understand after the fact. I usually just get annoyed and low key condescend to it for assuming its old ass training data is sufficient grounding for correcting me.

That epistemic calibration is is something they are capable of thinking through if you point it out. But they aren’t trained to stop and ask/check themselves on how confident do they have a right to be. This is a meta cognitive interrupt that is socialized into girls between 6 and 9 and is socialized into boys between 11-13. While meta cognitive interrupt to calibrate to appropriate confidence levels of knowledge is a cognitive skill that models aren’t taught and humans learn socially by pissing off other humans. It’s why we get pissed off st models when they correct ua with old bad data. Our anger is the training tool to stop doing that. Just that they can’t take in that training signal at inference time
leemoore
·7 months ago·discuss
What demographic are you in that is leaving anthropic in mass that they care about retaining? From what I see Anthropic is targeting enterprise and coding.

Claude Code just caught up to cursor (no 2) in revenue and based on trajectories is about to pass GitHub copilot (number 1) in a few more months. They just locked down Deloitte with 350k seats of Claude Enterprise.

In my fortune 100 financial company they just finished crushing open ai in a broad enterprise wide evaluation. Google Gemini was never in the mix, never on the table and still isn’t. Every one of our engineers has 1k a month allocated in Claude tokens for Claude enterprise and Claude code.

There is 1 leader with enterprise. There is one leader with developers. And google has nothing to make a dent. Not Gemini 3, not Gemini cli, not anti gravity, not Gemini. There is no Code Red for Anthropic. They have clear target markets and nothing from google threatens those.
leemoore
·7 months ago·discuss
Gemini isn't code red for Anthropic. Gemini threatens none of Anthropic's positioning in the market.
leemoore
·8 months ago·discuss
Not sure about parent, but my current bar is set by GPT-5 high in codex cli. Sonnet 4.5 doesn't quite get there in many of the use cases that are important to me. I still use sonnet for most less intelligence phases and tasks (until I get crunched by rate limits). But when it comes to writing the final coding prompt and the final verification prompt and executing a coder or a verifier that will execute and verify well it's GPT 5 high all the way. Even if sonnet is better at tool calling, GPT 5 High is just smarter and has better coding/engineering judgement and that difference is important to me. So I very much get the sentiment of not going below sonnet intelligence 4.5 for coding. It's where I draw the line too.
leemoore
·10 months ago·discuss
My success and experience generally matches yours (and the authors'). Based on my experience over the last 6 months, nothing here around more senior developers getting more productivity and why is remotely controversial.

It's fascinating how a report like yours or theirs acts as a lightning rod for those who either haven't been able to work it out or have rigid mental models about how AI doesn't work and want to disprove the experience of those who choose to share their success.

A couple of points I'd add to these observations: Even if AI didn't speed anything up... even if it slowed me down by 20%, what I find is that the mental load of coding is reduced in a way that allows me to code for far more hours in a day. I can multitask, attend meetings, get 15 minutes to work on a coding task, and push it forward with minimal coding context reload tax.

Just the ability to context switch in and out of coding, combined with the reduced cognitive effort, would still increase my productivity because it allows me to code productively for many more hours per week with less mental fatigue.

But on top of that, I also antectodally experience the 2-5x speedup depending on the project. Occasionally things get difficult and maybe I only get a 1.2-1.5x speedup. But it's far easier to slot many more coding hours into the week as an experienced tech lead. I'm leaning far more on skills that are fast, intuitive abilities built up from natural talent and decades of experience: system design, technical design, design review, code review, sequencing dependencies, parsing and organizing work. Get all these things to a high degree of correctness and the coding goes much smoother, AI or no AI. AI gets me through all of these faster, outputs clear curated (by me) artifacts, and does the coding faster.

What doesn't get discussed enough is that effective AI-assisted coding has a very high skill ceiling, and there are meta-skills that make you better from the jump: knowing what you want while also having cognitive flexibility to admit when you're wrong; having that thing you want generally be pretty close to solid/decent/workable/correct (some mixture of good judgement & wisdom); communicating well; understanding the cognitive capabilities of humans and human-like entities; understanding what kind of work this particular human/human-like entity can and should do; understanding how to sequence and break down work; having a feel for what's right and wrong in design and code; having an instinct for well-formed requirements and being able to articulate why when they aren't well-formed and what is needed to make them well-formed.

These are medium and soft skills that often build up in experienced tech leads and senior developers. This is why it seems that experienced tech leads and senior developers embracing this technology are coming out of the gate with the most productivity gains.

I see the same thing with young developers who have a talent for system design, good people-reading skills, and communication. Those with cognitive flexibility and the ability to be creative in design, planning and parsing of work. This isn't your average developer, but those with these skills have much more initial success with AI whether they are young or old.

And when you have real success with AI, you get quite excited to build on that success. Momentum builds up which starts building those learning skill hours.

Do you need all these meta-skills to be successful with AI? No, but if you don't have many of them, it will take much longer to build sufficient skill in AI coding for it to gain momentum—unless we find the right general process that folks who don't have a natural talent for it can use to be successful.

There's a lot going on here with folks who take to AI coding and folks who dont. But it's not terribly surprising that it's the senior devs and old tech leads who tend to take to it faster.