The SWE-Bench Illusion(microsoft.com)
microsoft.com
The SWE-Bench Illusion
https://www.microsoft.com/en-us/research/publication/the-swe-bench-illusion-when-state-of-the-art-llms-remember-instead-of-reason/
2 comments
It seems to be inevitable that metrics become targets and cease to be valuable.
This is a good reminder that benchmark results don’t always translate to real engineering work. Solving a scoped task inside a controlled setup is very different from working in a live codebase with missing context and messy history. Benchmarks are still useful, but they should be treated as one signal, not the full picture.