I wish the author had spent less time arguing why sublime is better or good at X Y and Z (and thus fall into the same tarpit as everybody else) and more that the importance-factor is entirely contingent on what the actual motivating factor, stakes, incentives, and outcomes really are.
Part of the problem is that online or in groups of varying skillsets/niches we lose that specific motivating context. So tools become ends unto themselves without the grounding that actually gives them value.
Because why should anybody give a shit about text editors at all? How I navigate and edit documents doesn’t have much material impact on the actual work that I’m trying to do. Maybe it does for others, or maybe it’s such a ubiquitous, low-stakes, context-free tool that it approaches a Platonic ideal topic for bikeshedding and flamewars.
What intrigues me is that that has a way of looping back around into popular culture and real economic/business decisions. You literally see it play out in real time with tech trends like mobile apps, Kubernetes, agentic tools, frontend frameworks, etc where they hit some critical combination of adoption and customizability, or conversely mature from a tinkerer/contrarian-oriented tool into one popular enough to be used for applications that require stability.
Is red better than lime green? Does red paint make cars go faster, or safer? Doesn’t matter what you think, if you sell fast cars, better have it in red
I think configurability depends on how important your tool is to the core job function or role being performed, where it becomes very valuable for helping them directly perform the tasks they and their employer value, vs how much it allows you make problems they don’t value as much get out of the way of the ones they do.
For example, I am a HUGE fan of the way Gusto handles payroll and all the different taxes and form filing for me, because I basically do not even have to think about the problem or fiddle with it at all. But to someone whose job is doing payroll/accounting/taxes or working within giant enterprise HR/legal/finance departments that does more harm than good, because it’s something they have to fight (or less charitably it makes their job too simple).
The other big problem is who is actually making the decision to pay or spend money on a thing, and whether it serves more of a defensive (eg auditability, security, constraints against undesirable behavior) or creative purpose. The creative stuff is sexier but hard to quantify, and end-users won’t actually be willing to pay that much for it relative to how much it helps them or how critical it is to their role.
It means the same thing to you, but not to the whole spectrum of people using AI. You literally see it on Reddit all the time where people are complaining about the same model either over-engineering or doing too much, vs it being requiring too much steering or not being autonomous or capable enough to hand off tasks to on its own.
The reason prompting it to review its own work for loose ends, record any new undocumented or noteworthy behavior, suggest changes to tests/processes to make it go more smoothly the next time, etc is that it’s prescriptive and process-oriented (and thus easily verifiable/done in-context) rather than descriptive and outcome oriented (which to do properly could require way more context than the model has, because it doesn’t know what it doesn’t know about your particular work, only what it’s seen so far).
Even promoting it to do these after-the-fact vs as an upfront requirement can have a big impact IMO. If you make “maintainability” part of the task before it’s seen the real work it will focus on general “best practices” crap rather than the real work, so either way if this is something you care about it doing you have to give it guidance for how you want it done.
If you were to review the logs of a model after the fact, you’d also not really save on input tokens unless you compressed the context or sharded it out, which can easily miss the small details that constitute the difference between “what actually happened” vs “how the LLM models this general class of problems” unless the first pass involves the entire context anyway. That said I do think there’s a lot of value in building some kind of pipeline for validating and aggregating these “learnings” across sessions.
You see this a lot with beginners, because until you’ve done the work long enough to truly know what works, you only really know what you have seen through other people’s performance of the work (to the degree it is even understandable and perceptible to you). Also your social circle is probably mostly other beginners or more experienced people who are evaluating you in terms of basic competency/understanding as someone who knows more than a guy off the street who wants to be or claims to be capable of something.
So the costly/difficult-to-fake signaling of competency through complex setups, or tool fluency, has very high personal value because it positions you as someone who is interested and capable of learning about this stuff. And if you don’t have any real work to do yet, or even know what it is all the work is actually done for, it’s the most obvious place to start.
Once you understand this you can start to understand how developer tools marketing actually works, and why “this completely eliminated that problem entirely!” is NOT what developers get excited about paying for or using unless it’s something they/their social peers don’t value. Conversely, if you create a vessel for them to participate in some kind of social trend/signaling game within their social world it stops mattering as much or not it’s more productive or doesn’t actually save any time.
This applies in almost all social systems, if you’re interested in learning more about it some good terms are “costly signaling”, “mechanism design”, and animal psychology. Just don’t let yourself think you’re too smart to do it yourself - it’s inherent to the act of socializing, so anytime you’re doing that, your perceptible behavioral signals are going to affect the outcome, whether you like it or not
I felt this. palpably after switching from big tech working on cloud infrastructure meant to be “invisible”/hands-off, to more devloper-tooling oriented products.
Humans are very interesting social creatures, we almost can’t help but participate in social signaling/status games and identity- building everywhere we go (and btw if your first inclination is a reaction or inclination to or respond “But I don’t! I’m an engineer, I stick to the facts!” you are literally acting that out yourself in real-time).
Signaling 101 is positioning/counter-signalling and establishing costly/difficult to fake proof to separate oneself out from the rabble; in that sense, products with difficult learning curves or which allow users to flex socially-valued skills within their social context are not really so much about what they do for that developer’s productivity, but for how they make the skill/value of that particular developer more apparent (I hate to say “legible” because it’s a claudeism) to other people. If I’m a random other developer how do I quickly decide how much social clout to award you as someone claiming to participate in the same social context? Well if you can probably demonstrate a difficult skill, like keyboard-only navigation, or delivering highly complex software, or maintain a very technical personal setup, you’re clearly “validated” as at-or-above baseline.
Then there’s another phenomenon where to a certain extent, all work done for other is performative if the intent is not purely altruistic/egoless. When you lack experience working in a field, all you know about it is what you’ve seen others perform. See: LinkedIn, random software projects from Reddit, SaaS built by get-rich-quick guys on lovable or indie hackers or whatever. But charitably, you have to start somewhere and how can you be blamed for not fully knowing what you don’t know?
This is why you see so many early beginners play around with neovim setups, installing random Linux distributions, arguing about which programming language is better, or writing medium articles about setting setting up Kubernetes. It’s not about doing the work - if you make their work look simple and easy you’re NOT helping them do what they want, they want a vessel through which they can demonstrate their own individual value and establish an identity other people value.
You see the same thing everywhere btw, Clay and all these openclaw products are the same thing for marketers, executives do it with emails, animals do it with frond and frills and antlers. Best to just accept it as part of life - unless you stop participating socially entirely, the “I’m better than that” view is just another social position you can either defend or something that gets in your way.
Just got it working with codex in a container! FYI I think there is a bug most others will run into at the Codex:Muse interface.
It's some kind of parsing or integration error due to what I think is codex not anticipating server-side tool calling and how meta treats those ids... first couple times running codex with muse, it would fail on its first non-web search call.
Got it fixed, not personally sold on the bespoke server-side tool calling and indefinite file storage yet, but also a very cool model that I'm enjoying using so far!
I’ve spent my time very similarly working on my own voice stack project, but having also seen how non-developers use AI or experience technology in general, I truly think they are better served with a different UX and product than what we have.
In other words, if you’re building your own voice inference tooling you’re just about the polar opposite user demographic than the one that truly needs and will value this. You’re using voice as a medium of convenience doing what existing models are technically and practically “shaped” to be able to do, knowing how they work well enough that conversation is more like typing/prompting with your voice than a natural interface. I’m guilty of this myself but have you ever even paid for a voice/audio model or hardware?
Compare that to the millions of people with an Alexa device in their home who buy products through it, or who prefer calling support to get a human over poring over technical documentation. They’re actually very close to finally getting a version of “Alexa” that lives up to its promise and I’m happy for them
I think the bare truth is that the target audience for this product is not people who are highly particular about terminology in answers involving vector mathematics.
It’s a different set of tradeoffs for users that don’t already have strong engagement or interest in existing AI products.
Do you realize how many more people prefer to chat over the phone and watch television or videos in their free time vs type multiple paragraphs of text into a chat window and then read 3x more back?
I’m not even talking about grandma here, it’s a non starter for the vast majority of humans who don’t spend their free time writing and reading tech news. To most people, having to write out a bunch of words describing their problem/goals, then sift through pages and pages of detailed response to get an answer, feels overwhelming and not worth doing.
This is something I built for myself, and to experiment with inference stacks. You can obviously just transcribe audio and hand it off to frontier models, so all you really need is a good voice stack and a “driver” for the interaction (like a phone call, place to see their work).
There are two big problems with this space IMO. One isn’t that you can’t get this to work but that people generally aren’t willing to pay for it for themselves, rather as a way to screen or automate stuff to be used by other people. Did you know Claude Code has a voice mode and that openai launched whisper a year ago, both of which have positive sentiment and adoption in heavy ai tool users? Yet it’s a blip in their marketing or why people use their products, meanwhile outside of coding, most of the biggest and highest earning AI product companies so far are voice agents targeting customer service, sales, business processes, etc.
The second is related: voice is genuinely a low-bandwidth medium, so as a primary interface for interacting with AI there is not a lot you can get out of it compared to eg complex technical work or visualizations or interactive applications. It is physically and mentally demanding to speak-aloud a highly detailed prompt fast enough that VAD won’t cut you off and you have something with comparable information density or specificity vs text. But to keep up a shorter and more natural cadence you’ll not be able to wait on a lot of thinking/tool unless you play UI tricks (ums and fillers, two models in a trench coat), break the illusion of a single coherent conversation, or take a lot of long pauses.
That’s why for the supplementary coding use case it’s mostly used for remote steering, and for general use marketed towards the large and very not-online group of people for whom typing is not a natural or common thing for them to spend their time on. Now that so much spend goes through heavily used token subscriptions and they’ve proven that kind of product, they’re not marketing “tool to get the most tokens per $ running your subscription 24/7” anymore lol.
What I’m most interested in is true “ambient” tool use against my own data or work, and for-later (or pushed live via your phone) visualizations or “five models in a trench coat but still coherent” UX, which you probably are too. But I think unless you work a lot with AI tools already it’s hard to understand how that’s any different from asking Alexa to set a timer, and either way something you’re not so desperate to have that you go looking for it, or pay smaller vendors/set up yourself.
I think it’ll just become “agentic search” and “information retrieval” again because RAG is too intertwined with a particular kind of implementation/use case of basic document scoring + first gen vector dbs that is IMO undesirable for more sophisticated approaches to associate themselves with.
You need a lot more unstructured data than most typical “RAG” users doing document search are dealing with for it it to not be a solved problem, IMO (just give a tool calling agent your sql schema/directory structure). Even that is still an interesting problem for more typical use cases, but only at large scales where you start needing to do multiple passes or fan-out or convert data that could be structured like that into data that already is. I’m interested in large scale code search, coding agent context/conversation search, and network/trace analysis which has a lot of domain-specific considerations that make it interesting but definitely not structured like a typical “document chunking with cosine similarity” RAG implementation.
To be honest as someone working in this space for the past two years, the problem with the “RAG” and semantic search community is it’s mostly vendors and solutions people selling simple, general stuff to product teams.
If you really are into search you probably implement something bespoke for your use case and integrate it into a product directly, and engage with models/infra tools directly rather than through the products in the space.
If you understand how “semantic retrieval” and other search tools are implemented in practice they feel almost embarrassingly primitive to give such fancy names, or pay for through tools that just implement really basic post-filtering. The entire space had the rug pulled out from under it once “agentic search” took off and most major LLM vendors started integrating web search and tool calling into their products. There is still a lot more interesting stuff you could do with customized rerankers/embedding models, and search algorithms, or small models specialized for agentic search/retrieval, etc but the userbase is big companies that realistically don’t need anything more than a list of tech support document titles that a cheap LLM can select from. So “RAG” is basically a sales shibboleth for that type of stuff now.
We really wanted to use sqlite-vec for this for our SSG but last we checked it hadn’t implemented HNSW/had good support for running vector search in-browser yet (I think it was still doing full-table scans?). I was pretty disappointed because after so many months/years, to not have that suggested to me that they weren’t up to task of delivering on their project, and I had recommended them as a worthy project for a grant I had also applied for, that they won and I didn’t.
If anybody knows of a good solution in this space, or if I’m wrong about SQLite-vec, please let me know. For our own SSG we’ve basically decided that we’ll give it a couple months while we work on other infra we want, then if they’re still not done we’ll just do it ourselves.
How do you test it across different workloads and are you running it in a datacenter or cloud provider?
I forgot to mention it but the other major problem I underestimated was giving the permission to potentially spend lots of money to AI calling each other in ways I didn't have a good way to monitor, and didn't want to actively watch. So I wanted to set budgets and have them get passed to children, and realized that meant I had to build a pretty complicated billing/scheduling system with a way to keep the part of it with all the permissions and money safe from the AI doing AI stuff on its own, and set up NAT and firewalls and all this other stuff.
If every child can loop back up to its parent, and everything can run stuff from the Internet, and make expensive resource decisions, and get restarted if it fails, then it might not ever converge on being done, or get infected or just mess up and spend a lot of money. I ask about the testing matrix/driver you're using because that's where I realized there was a lot of work and cost involved in getting that part working well enough to run real workloads.
IMO asking the AI to prompt itself, remind it to clean up other agents, tell it to monitor something and just hang there for hours, etc gets old really fast.
If I had to pay the API rate to have one LLM rewrite what I just told it to another one, then have the main one get busy or start waiting for subagents rather than be something I actively steer, and come back to the subagent being either gone or left hanging for hours blocking another one from doing the thing I actually asked it to do, I would never do it through Claude Code. It costs me only a few seconds to ask it do something and I almost never hit my usage limits without them, so I basically only use them because they're free.
For my own bulk workloads I just put codex and my own harness in container and built an API dispatcher for the repeatable workloads I care about. You can just pull from a queue or click a button or run a script, or use LLMs to launch them or review them, but it doesn't make any sense to me to have them "monitor" or manage each other passively because you just end up doing it anyway without a real API to control it.
My company tried to build something like this pre-TUI as a tool-AI-IO dag dispatcher. The biggest mistake I made was thinking that people would have no problem figuring out how they could translate their work or define multi-step automations, and focusing on the orchestration and sandboxing thinking that was the core, when it was really figuring out how to get the onboarding UX/complexity to not feel daunting or more trouble than it was worth.
Eventually for my own work, I discovered that the context management and runtime was more like a stream or active service mesh than a dispatching / one-off processing problem, most others' were too. Then all my prompts would degrade across model versions or providers, and I realized that actually setting the context for the tasks and keeping track of it all was a ton of work and something I had to do everytime as an actual user, but never when I was testing or demoing it on existing data.
Curious how you're testing your work and if you've managed to avoid the problems I ran into. I need to permute across the same set of workloads/configs you mention (and maybe more) for my next set of work so I'd be very interested in sharing or collaborating on the test infrastructure! At Google I did a lot of permutation testing using https://github.com/cloudprober/cloudprober and was going to start using it sometime in the next couple weeks. It exists basically one layer above the workload content/targets so it's probably compatible with everything except the test client/driver you're using.
We're working on a browser-harness that makes forking, rpcs, and mapreduce first class tool calling primitives. Among other things, this makes it easier to manage your own context, because you can visualize your agents, subagents, and active work and resources as they interact with each other across locally and remote environments. And it eliminates all the complexity of mcp and local sandboxing because that is literally the problem browsers were made to solve!
To be clear the browser IS the harness, it's not just a browser-based UI but also the sandbox and orchestration layer. By giving LLMs deep browser access (through CDP and some special hooks) they can verify their own UIs immediately after writing them, navigate the web natively, and run commands that directly manipulate the active DOM. This creates a very tight feedback loop for UI work, but also let's you create or run browser automations, or query a site by running a javascript query on its contents, or a web page without deploying or uploading it anywhere, which is pretty powerful. What I really like is that this makes it easy to dispatch cheap models to generate and verify tons of little visualizations using svg.
Locally it's just a browser, but to manage remote instances you can either access them as tabs on any local browser, or as inline collapsible iframes. I'm trying to be cautious with the security side of it so we're not marketing it as a product yet, but would love to work with some anybody who is interested and does a lot of UI or cloud work!
I'm excited about this particular moment in tech because I think work is going to end up looking like playing Starcraft with data and AI, surrounded by rich custom media as you work, which feels really futuristic to me!
Yes, that's the Internet Consensus and the reddit comment section every time a startup doing anything is mentioned.
You should never take a risk, business people are all evil and stupid, you should treat every employment or business opportunity as purely transactional because they'll do the same to you, there's nothing you can do about your job or employment, the only way to win is to cheat because everyone else is doing it, the key to happiness is educating other there's not really any cause and effect involved in the way things work unless you, personally, already know it. Just, you shouldn't do anything unless you understand everything about it, and if you don't it's not your fault.
> not flailing around is very difficult and unlikely
This is literally the defining trait of startups. What makes it stupid is that it's always more complicated than "engineer guy did everything he could but got screwed in the end" and that in real life, sometimes people do actually make money or establish businesses because of decisions they made, and conversely that there are real causes and effects behind things that don't go the way you want them to. Telling a story that doesn't contradict in anyway with consensus (so, directionally correct but always wrong) opinion has no point in the same way that there is no point telling a story where a knight rescues a princess by journeying through the kingdom making friends and overcoming challenges, then confronts the evil guy and kills him, the end. This is just that, but "the shady business guy and the screwed engineers"
I had the opposite reaction, this felt like a story that was literally purpose-built for pandering the hn audience without saying anything interesting.
Good fiction teaches you something you hadn't seen before, or challenges your perspective, or articulates a point of view or personality that you had never before considered. If it's just "some guy went to work and it sucked and he was right and everyone else was wrong and the Green People did classic Green People bullshit", and there's nothing else complicated or humanizing it, and no real-world lesson or stranger-than-fiction details to it, then what value does it have?
Like, what would happen if you asked a redditor with 10 years of experience reading about startups, but no real exposure to that culture/experience beyond the comment section, to write a story summarizing the consensus opinion on reddit of how startups typically work? Of course, because it's made up it's not wrong, but it exists entirely within the socially-contingent reality of the Internet Consensus.
In the real world there's politics, inter-personal relationships, personalities and personality flaws, and too much detail for "startup flails around" to be something you can reduce to "the startup flailed around". Of course it did, but why and how? A story that says "you know how it goes in all the other stories? yeah, that" or "there was a guy like you and he was good, and all the other guys were idiots and they were bad" has no point
I built a very similar tool recently mentioned elsewhere in these comments. I think with the current state of LLMs, harnesses, and related tooling, being able to create or setup self-eval tooling is the biggest differentiator between merely using LLMs to write code vs realizing true 10x productivity wins.
I'm curious whether this is something LLMs are eventually going to be good enough at doing, or something the average developer knows when and how to do, or if this is going be something that's too specialized or difficult for most developers and maybe the next generation of developer tooling products. Now that we're several months into Claude Code crossing the threshold of legitimacy and adoption, I've been surprised at how few projects or developers are doing this yet.
To a certain extent now all you need to do is ask Claude Code for browser automation workflows and CUJ tests in your repo, and ye shall receive, but probably something that just uses base playwright. It would be even better if you could ask to install or use a self-eval tool that already did everything you needed it to do and also knew how to specify/setup automations. I'm assuming the level of agency or mental overhead of embarking on a browser automation side quest is beyond what most developers are used to in the course of their regular work, even though it's not really as hard as it sounds now. If so then self-eval tooling could be a very promising new product category to sell to enterprises.
BTW if you have a link to your project I'd be interested in checking it out! $5.69 for a UAT run sounds very high to me based on how many tokens it typically takes for agents to create automations or steer my similar project, but it could be that your test workloads are much more exhaustive or high-dimensionality than mine are. This is what a basic "go to amazon.com and search for a product, then take a screenshot" automation looks like for ours: https://github.com/accretional/chromerpc/blob/main/recipes/s.... And this is our interactive/dynamic remote steering mode: https://github.com/accretional/chromerpc/blob/main/chrome-pr.... I decided to implement against the Chrome Devtools Protocol (one layer under Playwright) and use grpc service reflection to allow agents to dynamically discover/describe the entire chrome devtools api surface. I just started working on a way to gather traces and monitor/manage the automation run internals because I think there's a ton of opportunity in this problem space for orchestration and RL
I think most people who strongly identify with tools like vim do so out of a sense of identity-building to "be the kind of developer who is good at vim" / embody some kind of aesthetic or in-group signal moreso than an actual desire to be more effective at getting work done.
As long as you don't have some kind of stochastic or >5s impediment taking you out of a state of flow, most developers' productivity is going be vastly more influenced by their knowledge, understanding, and ability to focus on the problem they are working on than the marginal difference in time it requires to perform some navigation or editing task. Which is not to say that vim is bad or that you shouldn't use it, but that it's just a text editor and if you get triggered by someone not liking it or thinking it's more trouble than it's worth, it might be worth taking a step back and thinking about why it's something that triggers an emotional/defensive response, rather than the kind of reaction you'd have to someone liking strawberry more than vanilla.
Founder at Accretional (accretional.com). Building an agent mesh based on open source
Sponsor for statue.dev and previously at Google working on Serverless Infrastructure for Cloud Run and Cloud Functions
fred @ company or https://www.linkedin.com/in/fred-weitendorf-40b505b6/