That's an impressive jump in performance by providing the agent with access to relevant literature.
Is there a breakdown of which wins came from hyperparameter values (where BO would likely match this) vs. wins from techniques the agent wouldn’t have tried without the paper?
Trying this out on a valve engineering PDF extraction task — the repo claims 72% improvement on PDF extraction specifically, which is why I'm biting. Installed the MCP and the results were actually contextualized against what I'd already tried in the repo, which surprised me. Handing it to Claude Code now. Quick question: does it pull implementation details from the paper itself, or just the technique? Will report back.
You mention the index covers "applied CS" — where does it draw the line? If I'm implementing a time series forecasting model, does it cover the stats/econometrics literature too, or only CS-venue papers? A lot of the best techniques for engineering problems live in adjacent fields. Same question for computational biology, NLP-for-legal, etc.
Is there a breakdown of which wins came from hyperparameter values (where BO would likely match this) vs. wins from techniques the agent wouldn’t have tried without the paper?