🌐 WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation

Existing web agent evaluation suffers from three key limitations: (1) insufficient benchmark coverage — offline benchmarks lack real-world fidelity, while online benchmarks remain limited in website scale, domain diversity, and intent variety, leading to biased and overly optimistic assessments; (2) unscalable evaluation methods — human annotation is costly and impractical at scale, and current automated approaches lack sufficient accuracy on complex queries; (3) narrow evaluation dimensions — existing protocols fail to systematically assess agents' ability to leverage external knowledge and execute end-to-end workflows required for real-world deployment. WebRetriever addresses all three gaps through large-scale diverse benchmarking, a reliable automated evaluation method, and deployment-oriented evaluation protocols.

Current LLM-based evaluation pipelines exhibit limitations in organizing and utilizing evaluation signals. Existing methods employ three screenshot strategies: (1) full-trajectory screenshots, which provide comprehensive coverage but suffer from redundancy and high computational costs; (2) final screenshot only, which captures task completion status but misses critical intermediate process information; and (3) LLM-selected key screenshots, which reduce redundancy but lack sufficient accuracy in key step selection and matching. These limitations reveal the core challenge of automated evaluation: how to extract critical information to enhance LLM judgment accuracy. To address these limitations, we propose NavEval, which integrates more diverse information from agent-browser interactions. Beyond actions and final screenshots, NavEval leverages all navigation URLs and network requests generated during interactions. Through rule-based filtering to eliminate substantial noise, these signals are restructured into organized information that enhances LLM judgment accuracy.

To comprehensively evaluate web agent capabilities, we design three protocols reflecting realistic deployment scenarios. Protocol I assesses fundamental navigation ability by measuring whether agents can reach target pages given user tasks. Protocol II evaluates navigation performance when agents are provided with operational documentation. For Protocol II, operational knowledge is consolidated into structured documentation through a closed-loop framework. Web agents first generate action trajectories for target tasks, which NavEval evaluates automatically. Correct trajectories proceed to annotation, while incorrect ones are regenerated until reaching retry limits, after which human annotators refine them. The annotation platform further verifies accepted trajectories and streamlines redundant operations. Finally, LLMs generate operation manuals from these refined trajectories, completing the documentation pipeline. While Protocols I and II focus on navigation, realistic web environments require capabilities beyond page navigation. Agents must demonstrate deep page understanding, multi-source information integration, and precise extraction capabilities critical for holistic evaluation. Protocol III mirrors realistic deployment by assessing whether agents can accurately retrieve target information across multiple modalities (text, documents, charts).
Based on these designs, we collect 1,000, 1,000, and 100 tasks for Protocols I, II, and III, respectively. Notably, Protocols I and II share 550 overlapping tasks, with the sole difference being the availability of operational documentation.

To facilitate transparent and reproducible evaluation, we will publicly release the complete evaluation framework, including implementations of all three evaluation protocols, the NavEval automated evaluation pipeline, and human evaluation guidelines.