HackerTrans
TopNewTrendsCommentsPastAskShowJobs

cdetrio

128 karmajoined 11 năm trước
http://cdetr.io

comments

cdetrio
·14 ngày trước·discuss
I'd be very curious whether such junk lean code could be used to solve Kevin Buzzard's Jacobian challenge: https://gist.github.com/kbuzzard/778bc714030b3e974ab5f403878...

The idea behind the challenge to provide a few lines of human-written/checked scaffolding that defines an "API surface". The API surface is resistant to junk/slop filling out the sorry's. This is supposed to reduce the number of lines that humans have to read to only a handful, while the full valid AI-generated proof is tens of thousands of lines. The challenge was just solved a couple weeks, here's the discussion about it: https://leanprover.zulipchat.com/#narrow/channel/583336-Auto...
cdetrio
·3 tháng trước·discuss
Many parallels between this article and another posted today ("Unsubscribe from the Church of Graphs" https://news.ycombinator.com/item?id=47604253). Both are responses to astralcodexten posts, and the vibes felt around anecdata vs statistics.
cdetrio
·3 tháng trước·discuss
We found 500 zero-days in ten year old widely used open-source projects. Was that not a demonstration of the catastrophic failure of human debugging capability?
cdetrio
·4 tháng trước·discuss
The Duke harness was specifically designed for these puzzles, that's why they don't want to measure it.

My reading of that part in the technical report (models "could be using their own tools behind the model’s API, which is a blackbox"), is that there's no way to prevent it.

But from fchollet's comment here, using tools and harnesses is encouraged, as long as they are generic and not arc-agi specific. In that case, the models should be benchmarked by prompting through claude code and codex, rather than the through API (as from the api we only expect raw LLM output, and no tool use).
cdetrio
·4 tháng trước·discuss
Are you prompting the models through their APIs, which are not designed to use tools or harnesses? Or do the "system prompt" results come from prompting into the applications (i.e. claude code, or codex, or even the web front-ends)?
cdetrio
·4 tháng trước·discuss
Astrophysicist David Kipping had a podcast episode a month ago reporting that LLMs are working shockingly well for him, as well as for the faculty at the IAS.[1]

It's curious how different people come to very different conclusions about the usefulness of LLMs.

https://youtu.be/PctlBxRh0p4