peterbell_nyc

5 days agodiscuss

I would take the other side of this bet. While I agree that the impact of any given advance is likely to resemble a sigmoid curve, I think there is a material chance of "stacking sigmoids" creating something that looks exponential.

To take a simple example, look at the progress of technology over the last ~500 years - it seems to me that the rate of change continues to accelerate despite many of the logistic curves flattening along the way.

There are still huge unanswered questions about whether or not the stacking sigmoids will favor the incumbents. But I would not definitively bet against the people with the most compute data, talent and money.

peterbell_nyc

11 days agodiscuss

So many thoughts

AI can materially improve PRs - you just have to do it right.

Firstly there are not going to be a material number of non-AI patches written in the future. I know some people still ride horses, but when compared to the 17th century the percentage of travel primarily enabled by literal horse power is way down. Same way goes "artisinal, hand crafted code"

Secondly the problem isn't the AI - it's the crappy prompts and lack of intermediate artifacts and quality gates (deterministic and adversarial) in the generation pipelines.

Thirdly, don't disallow AI (I mean, feel free, you're doing something without charging for it - do you) - disallow crappy PRs, verbose descriptions. and bloated code that doesn't fit your coding standards.

Fourthly, ship your standards. Document what a good PR looks like, the comments and examples, the search for open PRs to ensure it isn't a dupe or a won't fix. Then ship your coding standards - have some LLMs infer them from your current code base and review (bonus - anything it infers correctly that you don't agree with, get it to re-code to your new standards).

Then set up a PR reviewer - start with deternministic gates for the format of the PR and some broad heuristics, then run it through adversarial reviews. Anything that looks great, feel free to run by a human either before the merge or after just to keep an eye on things.

The deterministic gates will cheaply get rid of 95% of the slop (AI and human - I've seem plenty of human slop over the years) so you can focus tokens on the good stuff.

peterbell_nyc

13 days agodiscuss

The problem isn't the AI - it's the assessments. It used to be that shipping a large, well-researched essay with multiple citations was proof of work. Now it's proof of prompting, not learning or effort.

If you want to test for memorization, you need a live test to do so.

If you want to test for comprehension and understanding, require sessions with a tutoring AI and grade the level of comprehension exhibited.

And then please, please test for prompting with challenging assignments that would usually be beyond a student's skills and make sure they know how to drive the machines to do valued work.

I totally agree that there is utility in memorization and comprehension. I also know that by the time students graduate, AI will be the job and if they can't pair successfully with agentic workflows they will not be much use in the work force for many roles.

peterbell_nyc

16 days agodiscuss

I auto tune my prompts to a locked model version based on production data used as evals with holdback data. I think the use case for this would be one off interactive prompts? For now I just run those all against an Opus 4.8 MAX and I'm sure I could downtune, although for interactive my opening prompt isn't always reflective of my overall goals for the multi turn session.

I'm just trying to figure out why on the fly routing would beat testing and tuning and locking models and versions for each class of call, with evals and auto tunes running to explore more possible models for commonly run classes of prompt over time . . .

peterbell_nyc

last monthdiscuss

Honestly my goal is to learn how to teach an agent to build a maintainable product, so I'm way more interested in the learnings at the agentic level (how to prompt/direct/manage context/restrict tool use, provide reusable shims, etc) than getting into the details of a css bug. That's just not a level of abstraction with sufficient leverage for what I'm trying to do.

I stopped coding a while back because I could have more impact directing a team of developers than writing code personally.

For my use case, the agents are now how I can have that scaled impact.

peterbell_nyc

last monthdiscuss

The agent isn't the hard part - it's the orchestration, skills, research systems, adversarial reviews, dreaming/compounding, context management and all the rest. Plus all the annoying hygiene tools to "poke the agent that got a clear prompt and decided to just sit there and wait for no good reason" and "delete the remote branches that the prompt told all the agents to delete but some of them forgot to":)

peterbell_nyc

last monthdiscuss

This this this!

Anyone have something they genuinely like for all of this? For now I'm rolling my own, but I can't believe I won't find a better OSS alternative soon...

peterbell_nyc

last monthdiscuss

Pi is a nice multi-agent wrapper. I use it to wrap my OpenAI max plan calls and my API calls. It takes care of some of the agent plumbing - still need sandbox, orchestrator, compounding, context, evals, etc but it's a nice component.

peterbell_nyc

last monthdiscuss

For me the heart of an agentic system is NOT using agents (except when you really have to). Components of a working system include: - Pipelines/recipes to describe multi-step flows (deterministic, agentic and HiTL steps), loops, conditionals, exit-on's for max loop iteractions, etc - The logistics to actually run the model and HiTL steps reliably across multiple agent worker pools - Management and delivery (and security/governance and permissioning) of thick skills with code to do as much as possible - Context management so the right agents have the right context for the right sessions at the right time - Project management - ability to store and access tickets, dependencies, track progress, restart stuck ticket claims, etc - Transcript saving, memory features and dreaming/compounding capabilities so the agents continue to learn from each session - o11y for understanding whats happening, tracking costs and usages, etc - Evals and auto-tuning of prompts so you can go cross model provider and also lock to a model version so you can do an ROI on each model version upgrade - Sandboxes for running the actual model sessions

Don't need to get it all from one vendor, but that feels to me like the toolkit and for most use cases I'd argue: - Don't limit yourself to a single model provider (anthropic, openai, etc) - Own your context - Own your compounding

peterbell_nyc

last monthdiscuss

Thank you. While I do not agree with him on every point (across his corpus - this one was excellent for it's time), it's a delight to re-read a Paul Graham essay.

There are a few topical short-form non-fiction writers that feel truly worth reading more than an agentic summary, and I get a visceral pleasure just following his words and logic to their well tuned conclusions.

Just reminds me how bad a lot of the AI accelerated content is these days. May have to few shot to improve my own writing :)

peterbell_nyc

last monthdiscuss

There's a huge difference between one shot and few shot versus building a robust harness with deterministic and adversarial quality gates. And I'm finding that agents can actually do a pretty good job of a surprising number of things if you are very clear about your dimensions of quality and the rubrics that you get agents to research and then use to validate against those dimensions of quality.

Make sure to use a deterministic pipeline or harness to go step by step so agents aren't checking their own work and I sometimes get alpha from having a codex check the work of a clod but I am seeing pretty good output across multiple domains when I have three independent quality gates and a loop which only spits it out to a human if it doesn't converge at a reasonable cost.

peterbell_nyc

last monthdiscuss

Anyone using this headlessly got a read on how much of this an agent could do without human intervention? Would love to have a gut check on "sure, spend the $295 and you'll get some benefits for free if you have an agent run your videos through this before shipping them"

To be clear, my use case is making weekly online videos suck a little less - not grading feature films :)

peterbell_nyc

2 months agodiscuss

Re: the higher ranking plagarism, that stings and makes sense. AEO and SEO are a thing. We need better mechanisms for identifying "root sources" of content - it's something I find myself working on personally. As I ingest sources for my book I need to be able to build a classifier that incrementally moves towards finding origin sources. That said, it's in my interest to do that because there is a differentiated value in having access to the sources that regularly provide novel, valuable content.

To be fair there is also value (at least for now) in sites that aggregate quality content and republish as a secondary level of discovery if my agents don't go far enough down the search results, but I'd expect that value to diminish over time as I better tune my research and build my lists of originating authors.

And to be clear, I don't like the idea of people stealing someone elses content and republishing without attribution (although it has been going on long before ChatGPT) but I think now we can all run agentic research teams the "bad actors" will slowly get filtered out of the ecosystem.

peterbell_nyc

2 months agodiscuss

I do just want to highlight that this is also what humans do. We read a bunch of content online and then use it in our work product. The vast majority of the value that I provide comes from copyrighted information that I have ingested - either directly with a payment to the creator (bought and read the book, paid for and attended the seminar) or indirectly via third party blog posts or summaries where I did not then pay the originator of the materials.

I think there are real questions around motivations for creation of novel, high quality valuable content (I think they still exist but move to indirect monetization for some content and paywalls for high value materials).

I don't inherently have any problems with agents (or humans) ingesting content and using it in work product. I think we just need to accept that the landscape is changing and ensure we think through the reasons why and how content is created and monetized.

peterbell_nyc

2 months agodiscuss

Love the DDD callout. I have explicit steps to review and rate delta's to the ubiquitous language and one of my architectural reviewers will often engage with me about where the bounded contexts should be and will probably the translation layers.

I find the more good practices I add to my envision/scope/spec/build/test/deploy loops the happier I am with the outcomes.

I will say that I am finding the actual code to be somewhat ephemeral for me - the more precise the specifications are and generally the tighter and more elegant the design is, the less the code matters as a long term artifact.

I'm not at the "code is assembler" point yet - but I could see that with more, richer specs I could end up there. Of course the specs are then substantial, but declarative specs can be robust and unambigous (with sufficient read teaming review) and - like domain specific languages - reduce the accidental complexity of the syntax when compared to an implementation in a given language.

There are exceptions to all of this, but it's fascinating to see how it's evolving!

peterbell_nyc

2 months agodiscuss

I'm generally in agreement with everyone here. - Some code is ephemeral - it's generated to do the thing, thrown away end of session and the csv was imported successfully (or whatever). Make sure you have at least some testing of the output or you may find the email is in the last name field for some rows. If possible, have an API your agent uses with rich domain types and validations that force it to do things right or do them again (and that it' can't rewrite to relax the constraints!) - You can one or few shot a real app - for a few users, for a small set of use cases. Scope of this will improve with models, but at least today it's spelling bee app for my kids" not "salesforce replacement for millions of workers". - You can add rich validation steps for all types of quality that you care about which (assuming they converge) can deliver high performance, well designed and functionally correct code mostly autonomously.

I'm building an orchestrator (who isn't). Haven't looked at the code yet, but it appears to work. But man have I spent hours in loops between Claude, Codex and myself all on the highest thinking levels to figure out what interface portability means for the employee, how best to handle "remote" sessions and the appropriate semantics for pipelines/recipes.

I've also been very opinionated about who does what. I'll let the agent write a script to sync with github and reload workers, but I decided to "waste" the 5 minutes to manually do all of the config steps on render for my server when claude told me that I couldn't just give it read only scope to pull the logs. Bad news, I'm cutting and pasting for my computer overlord. Good news? Claude can't blow away the prod db if it happens to get in the way of whatever interpretation is makes of the instructions I give it.

A chainsaw requires very different skills that an axe. It has different failure modes. Some experience as a lumberjack probably helps using either/both.

No difference (at least now) with agents.

peterbell_nyc

2 months agodiscuss

For me the distinction is the quality and rigor of your pipeline.

Vibe coding: one shot or few shot, smoke test the output, use it until it breaks (or doesn't). Ideal for lightweight PoC and low stakes individual, family or small team apps.

Agentic engineering: - You care about a larger subset of concerns such as functional correctness, performance, infrastructure, resilience/availability, scalability and maintainability. - You have a multi-step pipeline for managing the flow of work - Stages might be project intake, project selection, project specification, epic decomposition, d=story decomposition, coding, documentation and deployment. - Each stage will have some combination of deterministic quality gates (tests must pass, performance must hit a benchmark) and adversarial reviews (business value of proposed project, comprehensiveness of spec, elegance of code, rigor and simplicity of ubiquitous language, etc)

And it's a slider. Sometimes I throw a ticket into my system because I don't want to have to do an interview and burn tokens on three rounds of adversarial reviews, estimating potential value and then detailed specification and adversarial reviews just to ship a feature.

peterbell_nyc

2 months agodiscuss

Helps if you both hand to original agent as strong guidance and then to an adversarial agent as a quality reviewer. The adversarial agent is more likely tro loop the work back if it fails the validation criteria.

I do find that just asking the same agent to do and check it's own work is not particularly reliable.

peterbell_nyc

3 months agodiscuss

Why crack one website when you can crack all of them? For a well funded (especially nation state) attacker, if $1 in compute and effort returns $2 in ransoms, when it's possible to access another n x $1 of compute and if you don't hit diminishing returns or cashflow limitations, why wouldn't you just keep spending $'s until you p0wned all the systems?

If there is only one bear, you just need to run faster than your friends. If there's a pack of them, it you need to start training much harder!

peterbell_nyc

3 months agodiscuss

Exactly this. I'm writing my own little orchestrator and memory system and because I have a modest number of workflows, I'm taking the time to specify them deterministically, describe them as a DAG (with goto's for the inevitable loops) and generate deternministic orchestration code. I'm trying to make most of the tool calls as clear and comprehensive as possible (don't make Opus convert a PDF, have a script do that and give it the text instead) and I'm putting all the things you'd expect to track state and assume ~20% task failure rate so I can simply wipe and repeat failed tasks.

Small model and (where still required) human in the loop steps for deterministic workflows can solve a surprisingly large number of problems and don't depend on the models to be consistent or not to fail.

Just invest heavily in adversarial agents and quality gates and apply transforms on intermediate artifacts that can be validated for some dimensions of quality to minimize drift.