I am interested in why you chose to do this, and publish it with the headline you used. Was it to learn something? Or to get publicity for another project?
Tbh, this sounds like fear mongering to me. Of course the statement “99.9% of servers are not compliant” sounds impressive, but then it turns spec hasn’t even been released yet.
Also some general feedback: the whole thing looks generated, as does the comment I am replying to.
It’s personal ad, basically. The author is trying to get a job as an evaluator somewhere and is hoping that putting 1000$ on the line will get them enough publicity to land them an interview/get a job somewhere.
The second part of this comment is not what I expected. I also don’t think it is true. I got bit by a CORS error at work recently that passed by Claude, copilot, and another senior engineer.
We’ve been on the receiving end of this complaint with Semble. I think it is a valid complaint, but constructing a benchmark for this kind of thing is just very difficult and expensive because of the (harness) x (model) x (mcp/cli) combination.
With traditional ml/tooling, not showing benchmarks was usually a red flag. But for llm tooling, I’m not so sure.
Thanks! This is very similar indeed. Related: I see a lot of “drive-by” PRs by agents, who obviously have no intent of ever maintaining the code they wrote.
I’m not sure I share your view of PRs. I still see submitting PRs as something that puts pressure on maintainers. Even incorrect PRs take time to verify and review.
I also don’t see how this differs between the “gap” and the “fence” part of the metaphor. Whether someone submits a rewrite/removal (fence) or a new feature (gap) for PR review, it’s still going to cost me attention.
It’s an interesting question: I’d say this is more of a vulnerability creator than the actual vulnerability.
Similar to how using very difficult technologies makes you more likely to create code with vulnerabilities: the technologies are not the vulnerability, but it’s easier to cause them.
This paper oversells on the title. Like, what is chronos, which embedding model was used, which reranker, how was the reranking done, why is chronos much better than claude code
Sure, the whole premise is exactly that proof of work reduces the value of scraping, while having negligible impact on users. If the data is so valuable that bot operators are willing to pay 10s of cpu, then other measures are necessary.
Nevertheless even for these high value cases, you can still argue that it disincentivizes the business model, it becomes less efficient.
I was also at the event and was pretty disappointed. Most of the talks were pretty low on information. I was at the “build” stage, which supposedly was the technical stage, but the talks there didn’t really go into technical specifics.
It was directed at the parent who implied that we didn’t think about this.
I agree with your point about the evals and how you can get discontinuities: good search can be worse than bad search when agents can do many searches. We’re working on it