DatasetAction history improves web-agent decisions
Across five models, including previous actions improves performance on AitW and MONDAY. The figure shows average accuracy with and without action history.
AgentVQA
A Unified Benchmark for
Agentic Visual Understanding
Abstract
Vision-language models (VLMs) can perform a broad range of tasks across diverse settings, yet their performance in agentic contexts remains poorly understood. Although online evaluations in simulators and real-world environments are essential for measuring interactive agent performance, they are costly and challenging to scale. Complementary offline VQA benchmarks can enable faster and more reliable comparisons across agentic domains, yet existing benchmarks remain limited and fragmented. To address this gap, we introduce AgentVQA, an offline benchmark for evaluating VLMs on realistic agentic scenarios across five agent domains: Web Agents, Robotics, Egocentric Videos, Games, and Spatial Planning. We build AgentVQA by manually selecting 1000 challenging, real-world agent examples from 23 source datasets and adapting them for reliable offline evaluation. The resulting benchmark preserves realistic scenarios from interactive environments across diverse domains, and its model rankings track those from corresponding online evaluations. Across the 22 evaluated models, the best achieves only 58.5% accuracy. Our cross-domain analysis shows that errors are driven primarily by failures to ground and maintain accurate representations of the current and evolving visual state, including localizing relevant UI elements, interpreting motion, tracking state changes over time, and inferring 3D spatial relationships, rather than by task misinterpretation or downstream logical reasoning.
Examples
One example from each of the 23 datasets. The correct answer is highlighted; model symbols indicate predictions.
Dataset curation
Questions undergo MCQ conversion, VLM screening, and manual checks for correctness, clarity, difficulty, and distractor quality.
The final benchmark retains 22% of initial candidates. Each retained question is verified to have exactly one correct answer.
Manual review (Stage 4) catches ambiguities that automated screening can miss. In the left example below, the scissors’ bounding box includes background, so a click inside the box can be scored as correct even if it misses the scissors.
The right example has more than one plausible next action. Clicking the Lyft icon (B) opens the app, but opening the app drawer (D) and then launching Lyft also leads toward deleting the account, with one extra step. Because the question does not require the shortest route, both actions are defensible. We remove or repair such questions to ensure a single unambiguous correct answer.
Model performance
The best overall accuracy is 58.5%. No single model leads across all five domains.
Rank correlations
We compare model rankings on AgentVQA with general reasoning benchmarks and an online agent benchmark.
A different ordering from general benchmarks
The paper evaluates matched subsets with standardized prompts. Model rankings shift across MMMU-Pro, GPQA-Diamond, and ChartQA-Pro, with moderate to weak rank correlations. These differences suggest that AgentVQA provides a complementary view of visual decision-making capabilities that general benchmarks may not fully capture.
MMMU-Pro0.67
GPQA-Diamond0.59
ChartQA-Pro0.44
Spearman’s ρ · 10-model comparison
Strong agreement in a matched online domain
AgentVQA Spatial Planning and the SpatialWorld Physical track agree strongly across their 12 shared models. This is a separate overlapping-model cohort from the main 22-configuration evaluation. This agreement supports AgentVQA as a lower-cost offline proxy for comparing models in spatial planning, capturing ranking differences observed in online evaluation without requiring environment setup or repeated agent rollouts.
ρ = 0.89512 shared models · p = 8 × 10−5
Ablations
Reasoning gains depend on the model
Extended reasoning improves both GPT-5.6 variants, but GLM-4.1V-9B-Thinking trails its base counterpart. Accuracy alone does not tell the full story of how models use extra computation.
| Model | None / Base | High / Thinking | Δ (pp) |
|---|---|---|---|
| GPT-5.6-sol | 38.2 | 55.0 | +16.8 |
| GPT-5.6-luna | 32.8 | 47.9 | +15.1 |
| GLM-4.1V-9B | 36.3 | 35.4 | −0.9 |
Δ is the difference in percentage points. The paper also documents “thinking loops” in GLM-4.1V-Thinking.
Error analysis
The paper analyzes incorrect Gemini 3.1 Pro predictions sampled approximately evenly across datasets. Error categories are model-assisted and manually verified.
| Domain | Most common error | Share of analyzed errors |
|---|---|---|
| Web Agents | Grounding error | 78% |
| Egocentric Videos | Motion misperception | 39% |
| Robotics | Spatial confusion | 65% |
| Games | Visual miscomprehension | 71% |
| Spatial Planning | Viewpoint gap | 62% |
Percentages are shares of the analyzed errors within each domain—not error rates across all questions.
Citation
If you find AgentVQA useful in your research, please consider citing:
@misc{gupta2026agentvqaunifiedbenchmarkagentic,
title={AgentVQA: A Unified Benchmark for Agentic Visual Understanding},
author={Advait Gupta and Sachin Konan and Tom Yin and Gabriel Sarch and Zhuang Liu},
year={2026},
eprint={2610.31494},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2610.31494},
}