BrowseComp is a web browsing benchmark, not a knowledge or reasoning test. It evaluates whether AI agents can navigate the open web to find specific, obscure information.
Questions are “inverted” - authors start with a fact and work backwards to create a question that’s easy to verify but extremely hard to solve through search.
Brute-force search doesn’t work. The search space is deliberately massive - thousands of papers, matches, events - making systematic enumeration impractical.
Grading uses an LLM judge with a confidence score, creating an interesting meta-layer where one model evaluates another’s certainty.
This benchmark reveals the gap between “can answer questions” and “can do research” - the exact capability that separates chatbots from useful AI agents. Built-in planning loops (TodoToolset)
Filesystem access and file upload handling
Subagent delegation
Extensible skills system (define new behaviors with simple markdown prompts)
Multiple state backends: in-memory, persistent filesystem, secure DockerSandbox, and CompositeBackend
Automatic conversation summarization for long sessions
Human-in-the-loop confirmation workflows
Full streaming support
Native structured outputs via Pydantic models (output_type)
Key features list: Multiple Backends: StateBackend, FilesystemBackend, DockerSandbox, CompositeBackend
Rich Toolsets: TodoToolset, FilesystemToolset, SubAgentToolset, SkillsToolset
File Uploads: run_with_files() and deps.upload_file()
Skills System: markdown-based skill definitions
Structured Output: type-safe Pydantic responses
Context Management: auto-summarization
Human-in-the-Loop: built-in approval steps
Streaming: token-by-token responses
There’s a complete demo app in the repo that shows streaming UI, file uploads, reasoning traces, and human confirmation flows:
https://github.com/vstorm-co/pydantic-deepagents/tree/main/e... # Run the agent
result = await agent.run("Help me organize my tasks", deps=deps)
print(result.output)
asyncio.run(main())
``` Multiple Backends: StateBackend (in-memory), FilesystemBackend, DockerSandbox, CompositeBackend
Rich Toolsets: TodoToolset, FilesystemToolset, SubAgentToolset, SkillsToolset
File Uploads: Upload files for agent processing with run_with_files() or deps.upload_file()
Skills System: Extensible skill definitions with markdown prompts
Structured Output: Type-safe responses with Pydantic models via output_type
Context Management: Automatic conversation summarization for long sessions
Human-in-the-Loop: Built-in support for human confirmation workflows
Streaming: Full streaming support for agent responses