Training is expensive so I wouldn't necessarily call it "pragmatic".
- the tool's goal is actually to provide a lightweight, practical way to avoid wasting training cycles on bad data.
Evals for robotics are also expensive.
- validation loss is a poor proxy of robot performance because success is underconstrained by imitation learning data
- most robot evals today are either done in sim (which at best serves as a proxy) or by a human scoring success in the real world (which is expensive).
It's great if you have evals and want to backtrack (we're building tools for that too) but you definitely don't want to discover you have bad data after all that effort (learned that the hard way, multiple times).
The metrics the tool scores vary from tedious to impossible for a human to sanity check so there's some non-obvious practical value in automating some of it.
The nature of changes is that you cannot predict them in advance.
Even if you knew your roadmap for 2 months and were operating perfectly, something unpredictable could happen, as things do. Such things don’t give you any kind of notice before they happen.
In such an event, you can’t be expected to remember something someone told you 2 weeks ago in passing anyway. Standups are a short but heavy information dump and are not good at facilitating recall as time goes on. This is true especially if you are operating in a rapidly changing environment such as a startup.
Except that a stand up doesn’t actually solve the problem you are referring to. People don’t pay much attention to what you’re working on in stand ups since it’s most probably not relevant to what they’re working on at the time. It is unreasonable to expect people to have perfect foresight that what you’re working on is going to be relevant to them 2 weeks down the line. There are better ways to solve that problem but stand ups is not one of them.
- the tool's goal is actually to provide a lightweight, practical way to avoid wasting training cycles on bad data.
Evals for robotics are also expensive.
- validation loss is a poor proxy of robot performance because success is underconstrained by imitation learning data
- most robot evals today are either done in sim (which at best serves as a proxy) or by a human scoring success in the real world (which is expensive).
It's great if you have evals and want to backtrack (we're building tools for that too) but you definitely don't want to discover you have bad data after all that effort (learned that the hard way, multiple times).
The metrics the tool scores vary from tedious to impossible for a human to sanity check so there's some non-obvious practical value in automating some of it.