I have written a couple of eval harnesses to see how well LLMs drive software I have written. Basically I have data analysis software that I need LLMs to write code for. The code is complex, and I want to shape my APIs such that LLMs do a better job of quickly getting to the right answer. So I test different prompting and api surfaces, it's really easy to make quick gains this way and save your users from bugs. In this paradigm, I'm explicitly not testing different models, and I'm very interested to see how lesser models do with my software. Also for this type of testing, using the open weight models makes it faster, cheaper, and more reliable to test vs frontier models because I can trust that kimi-2.5-a-bunch-of-specs is going to behave more consistently than whatever tweaks Claude is making to Sonnet this week. API and prompting improvements seem to carry across the different models for gross improvements.
I haven't looked that hard, but I can't find articles about this type of eval testing, curious to hear if others have approached writing APIs in this way.
I'm as anti-car and pro-bike as they come. Cars and trucks are a much bigger danger than e-bikes...
But California's clear tiers of ebike regulations are meaningless without enforcement. Over the past half decade blue states have become unwilling to enforce almost any laws. when they do enforce the laws it is sporadically. This matters for ebikes, it matters more for cars. Running a stop sign is absolutely not enforced any more.
Python/Jupyter developer in Boston. I build data tools and I start by talking to the people who'll use them, because code that doesn't get adopted is wasted work. My main thing is Buckaroo (github.com/paddymul/buckaroo, ~680 stars), an open-source data table for Jupyter over Pandas/Polars. I built both the data layer and the React frontend. Looking for a team building data tooling (as a product or in house).
Ted Turner won the America's cup there in 1977. His team named Courageous was legendary. Robbie Doyle was a team member, and got a degree from Harvard in applied physics. In the middle of the trials to see which team would defend the cup for the US, he remade the sails to be more competitive. Doyle went on to found a racing sailmaking company.
I used to live in Newport, RI. I love sailing and introducing people to the world of sailing. When I had guests I asked them to watch this NBC video about Ted's 77 campaign [1]. It really captures the history of Newport, sailing, and Ted
Ted Turner won the America's cup there in 1977. His team named Courageous was legendary. Robbie Doyle was a team member, and got a degree from Harvard in applied physics. In the middle of the trials to see which team would defend the cup for the US, he remade the sails to be more competitive. Doyle went on to found a racing sailmaking company.
I used to live in Newport, RI. I love sailing and introducing people to the world of sailing. When I had guests I asked them to watch this NBC video about Ted's 77 campaign [1]. It really captures the history of Newport, sailing, and Ted
I think another factor is that people are rejecting the rounded corners and excessive padding of modern web design, you can't do that in a TUI, so you don't have a designer or standard practice encouraging you to do it. As implemented TUIs have greater information density than GUIs. Make no mistake though, TUIs are a decided step backwards from GUIs. Everything that you can express via text, you can also do in a text area on a GUI app.
Slightly related to the article. I have a personal cargo bike. The most fun that I have with it is giving friends a ride home from a party. People instantly start giggling and laughing. It's goofy, you get stares and people curious
I'm not concerned about making money from it, I just want to use it. I'd like to check to see if I'm re-inventing the wheel. I'm curious if others would like a similar experience.
I want a system that enforces planning, tests, and adversarial review (preferably by a different company's model). This is more for features, less for overall planning, but a similar workflow could be built for planning.
1. Prompt
2. Research
3. Plan (including the tests that will be written to verify the feature)
4. adversarial review of plan
5. implementation of tests, CI must fail on the tests
6. adversarial review verifying that the tests match with the plan
7. implementation to make the tests pass.
8. adversarial PR review of implementation
I want to be able to check on the status of PRs based on how far along they are, read the plans, suggest changes, read the tests, suggest changes. I want a web UI for that, I don't want to be doing all of this in multiple terminal windows.
A key feature that I want is that if a step fails, especially because of adversarial review, the whole PR branch is force pushed back to the previous state. so say #6 fails, #5 is re-invoked with the review information. Or if I come to the system and a PR is at #8, and I don't like the plan, then I make some edits to the plan (#3), the PR is reset to the git commit after the original plan, and the LLM is reinvoked with either my new plan or more likely my edits to the plan, then everything flows through again.
I want to be able to sit down, tend to a bunch of issues, then come back in a couple of hours and see progress.
I have a design for this of course. I haven't implemented it yet.
I’m not running fleets of gpus so I’m not your target market, but can you tell me more about who is doing this type of large scale training outside of companies like anthropic, OpenAI, xai…. The big teams. I’m just curious?
I am getting really tired of github. outages happen that's a given. but on so much stuff they don't even care or try. Github is becoming the bottleneck in my agentic coding workflows. unless I make Claude do it intelligently, I hit rate limits checking on CI jobs (5000 api requests in an hour). Depot makes their CI so much better, but it is still tied to github in a couple of annoying places.
PRs are a defacto communication and coordination bus between different code review tools, its all a mess.
LLMs make it worse because I'm pushing more code to github than ever before, and it just isn't setup to deal with this type of workload when it is working well.
It helps with latency too or schedule padding. Bus schedules are unreliable because of all the stops which slow them down and encourage bunching of busses on a route with a lot of service.
That's a great idea, and I was just thinking about how it would pair with self hosted CI of some type.
Basically what I would want is write a commit (because I want to commit early and often) then run the lint (and tests) in a sandboxed environment. if they pass, great. if they fail and HERAD has moved ahead of the failing commit, create a "FIXME" branch off the failure. back on main or whatever branch head was pointed at, if tests start passing, you probably never need to revisit the failure.
I want to know about local test failures before I push to remote with full CI.
automatic branching and workflow stuff is optional. the core idea is great.
I haven't looked that hard, but I can't find articles about this type of eval testing, curious to hear if others have approached writing APIs in this way.