Show HN: New Benchmark from SWE-bench team is 0% solved(programbench.com)
programbench.com
Show HN: New Benchmark from SWE-bench team is 0% solved
https://programbench.com/
4 comments
This looks pretty interesting, but I don't understand why decompilers are not allowed. If this benchmark was aimed at recreating a SASS/server based product then it might make more sense, but given the fact that decompilers are readily available in practice the "no read" restriction seems to artificially increase the challenge level.
figures 10 and 11 in the paper are interesting.
i suppose at a high level this works because it is much easier for the evaluator to generate tests with fuzzing than it is for the model to probe.
this method somehow clarifies the way in which code generation is curve fitting, where the output curve is some linear transformation of the inputs.
kind of satisfying that when all is said and done, and we have a machine that can fit curve descriptions as well as or better than humans, we won't be any closer to explaining how anything really works.
i suppose at a high level this works because it is much easier for the evaluator to generate tests with fuzzing than it is for the model to probe.
this method somehow clarifies the way in which code generation is curve fitting, where the output curve is some linear transformation of the inputs.
kind of satisfying that when all is said and done, and we have a machine that can fit curve descriptions as well as or better than humans, we won't be any closer to explaining how anything really works.