Large multimodal models (LMMs) have exhibited proficiencies across many visual tasks. Although numerous well-known benchmarks exist to evaluate model performance, they increasingly have insufficient headroom. As such, there is a pressing need for a new generation of benchmarks challenging enough for the next generation of LMMs. One area that LMMs show potential is graph analysis, specifically, the tasks an analyst might typically perform when interpreting figures such as estimating the mean, intercepts or correlations of functions and data series. In this work, we introduce GRAB, a graph analysis benchmark, fit for current and future frontier LMMs. Our benchmark is entirely synthetic, ensuring high-quality, noise-free questions. GRAB is comprised of 2170 questions, covering four tasks and 23 graph properties. We evaluate 20 LMMs on GRAB, finding it to be a challenging benchmark, with the highest performing model attaining a score of just 21.7%. Finally, we conduct various ablations to investigate where the models succeed and struggle. We release GRAB to encourage progress in this important, growing domain.

翻译：大型多模态模型（LMMs）已在众多视觉任务中展现出卓越能力。尽管目前存在诸多知名基准用于评估模型性能，但其性能提升空间日益受限。因此，亟需构建足以挑战新一代LMMs的新一代基准。图分析是LMMs展现潜力的重要领域，具体涉及分析人员在解读图表时通常执行的任务，例如估算函数与数据序列的均值、截距或相关性。本研究提出GRAB——一个适用于当前及未来前沿LMMs的图分析基准。该基准完全通过合成生成，确保问题具有高质量、无噪声的特性。GRAB包含2170个问题，涵盖4类任务和23种图属性。我们在GRAB上评估了20个LMMs，发现其具有显著挑战性，性能最优模型的得分仅为21.7%。最后，我们通过多项消融实验探究了模型的优势与薄弱环节。我们公开GRAB基准以促进这一重要且持续发展的领域取得进展。