Large models have recently played a dominant role in natural language processing and multimodal vision-language learning. However, their effectiveness in text-related visual tasks remains relatively unexplored. In this paper, we conducted a comprehensive evaluation of Large Multimodal Models, such as GPT4V and Gemini, in various text-related visual tasks including Text Recognition, Scene Text-Centric Visual Question Answering (VQA), Document-Oriented VQA, Key Information Extraction (KIE), and Handwritten Mathematical Expression Recognition (HMER). To facilitate the assessment of Optical Character Recognition (OCR) capabilities in Large Multimodal Models, we propose OCRBench, a comprehensive evaluation benchmark.Our study encompasses 29 datasets, making it the most comprehensive OCR evaluation benchmark available. Furthermore, our study reveals both the strengths and weaknesses of these models, particularly in handling multilingual text, handwritten text, non-semantic text, and mathematical expression recognition. Most importantly, the baseline results showcased in this study could provide a foundational framework for the conception and assessment of innovative strategies targeted at enhancing zero-shot multimodal techniques. The evaluation pipeline and benchmark are available at https://github.com/Yuliang-Liu/MultimodalOCR.

翻译：近期，大型模型在自然语言处理和多模态视觉语言学习中占据主导地位，但其在文本相关视觉任务中的有效性仍相对未充分探索。本文对GPT4V、Gemini等大型多模态模型在多种文本相关视觉任务（包括文本识别、场景文本中心视觉问答（VQA）、文档导向VQA、关键信息提取（KIE）及手写数学表达式识别（HMER））中进行了全面评估。为便于评估大型多模态模型的光学字符识别（OCR）能力，我们提出了综合性评估基准OCRBench。本研究涵盖29个数据集，是当前最全面的OCR评估基准。研究揭示了这些模型的长处与短板，尤其在多语言文本、手写文本、非语义文本及数学表达式识别方面。最重要的是，本研究展示的基线结果可为构思和评估创新策略提供基础框架，以增强零样本多模态技术。评估流程与基准详见https://github.com/Yuliang-Liu/MultimodalOCR。