Second comment, having read in more depth (really love the auto-layout detail!) - the spec doesn't seem to naturally support layering (which is useful in some multi-axis automatic cases) - any plans for composability?
Semantic types as the extra formatting factor is super useful because they are concise encodings of a lot of formatting boilerplate.[1] Do you envision the flint type registry being shared/extensible? Why not have that be a data property itself?
[1] landed on pretty much the same spec for higher level tether charting on vegalite
Very much in agreement with this but especially on the UX side I find that 'what is intuitive to me' is not universal because you come with a bunch of contextual priors you can't clear. I wish there was more tooling focused on how to use tools to expand customer empathy past what you can directly experience; that's an inherently audience-limiting approach.
(I admire PMs that can cross that chasm; it's also why they're often bad at infra tooling, because you can just short-circuit the empathy).
Exactly, so it has a success rate of 0 and infinite cost/completion on your relevant benchmark. If the benchmark doesn't map to what you need it to, then yeah, it's not a useful input.
If you don't have a backend, then it's all telemetry, right? And backend logs don't capture a lot of the UX side of things - how a call got triggered, from where, etc (which yeah you can start to instrument, but then that's telemetry).
It feels like you could argue that since you control nature/nurture it's very possible to create a model aligned to an arbitrary spec - there is no theoretical reason it's not possible given N runs, and you only need to take the successful one. (ethically very.. questionable in humans) I think it's much trickier to define that spec, much less measure it and validate that a model is aligned to it.
I'm not sure the security/safety stuff is entirely in their control at this point, though you can argue that they are indirectly responsible through encouraging regulation via their positions on safety/risk.
I find it interesting that when drawing this parallel you mention that some devs 'get it' and 'build a great culture'; I think this is exactly where the analogy breaks down. Good managers get great results from people (and for people! they are linked).
Good AI managers are just running optimization loops at more declarative levels. Yeah, you need to get comfortable with less personal review of code for both, but I think the differences outweigh the commonalities - it's much easier for someone with a more 'traditional' IC model to be successful with agents then they would be with management, and I think most (good) management training would be entirely irrelevant. Parallels are maybe tighter to higher IC progressions.
I think it often useful to push the conversation down "we built a system for humans that dealt with this, what from that is or is not applicable for agents in the same context"? Humans randomizing resume review for screening is pretty known; I've seen companies try to fight it with things like hiding information, panel reviews, etc - it's unclear to me how effective those would be for agents (honestly, it was unclear how effective those were for humans). I was depressed about the hiring process before we had AI screening and I remain depressed about it.
This starts to sound more like ‘social engineering a human assistant’, so there’s a degree of required specialization that does meaningfully increase costs.
I think the theory is that it’s not purely static - you need to keep training and tuning the model (even just for general knowledge upkeep with current architecture) and so the infra/data is a contributory moat. Exfiltrating weights would get you a depreciating asset (plus we have all the lovely legal and regulatory frameworks to further protect them, which IS more like publishing)
The SELECT machinery is the product with databases! SQL often the shortest description of the processing logic, and the database has an efficient local execution engine that can prune/reduce data read based on the plan. Very hard to match in app, especially when joins get involved.
I’d be interested in the benchmarking if you ever write it up! People do seem to assume LLM as a judge/panel improves outcomes (and arguably it does in cases like code review?) but I suspect it is very situational and the priors from human panel of experts don’t always translate cleanly.
Been working on optimizing CLIs for cheap agent use and figuring out how to build integrated agentic features that aren’t a full chat interface. Agent UX optimization is kind of fun! Much more testable than human UX, though it’ll be interesting to see how much generalizes across model families.
Been doing this to improve/simplify the grammar for Trilogy[1], a streamlined SQL language - I’ve been planning a redo of one feature and it’s nice to be able to rapidly get feedback on various syntax success rates. Also been particularly useful to optimize error messages, which should help people too.