Large models have recently played a dominant role in natural language processing and multimodal vision-language learning. It remains less explored about their efficacy in text-related visual tasks. We conducted a comprehensive study of existing publicly available multimodal models, evaluating their performance in text recognition (document text, artistic text, handwritten text, scene text), text-based visual question answering (document text, scene text, and bilingual text), key information extraction (receipts, documents, and nutrition facts) and handwritten mathematical expression recognition. Our findings reveal strengths and weaknesses in these models, which primarily rely on semantic understanding for word recognition and exhibit inferior perception of individual character shapes. They also display indifference towards text length and have limited capabilities in detecting fine-grained features in images. Consequently, these results demonstrate that even the current most powerful large multimodal models cannot match domain-specific methods in traditional text tasks and face greater challenges in more complex tasks. Most importantly, the baseline results showcased in this study could provide a foundational framework for the conception and assessment of innovative strategies targeted at enhancing zero-shot multimodal techniques. Evaluation pipeline will be available at https://github.com/Yuliang-Liu/MultimodalOCR.

翻译：近期，大型模型在自然语言处理和多模态视觉语言学习中发挥了主导作用，但其在文本相关视觉任务中的效能仍待深入探究。我们系统研究了现有的公开多模态模型，评估了其在文本识别（文档文本、艺术文本、手写文本、场景文本）、基于文本的视觉问答（文档文本、场景文本及双语文本）、关键信息提取（收据、文档与营养标签）以及手写数学表达式识别方面的表现。研究结果揭示了这些模型的优势与不足：它们主要依赖语义理解进行单词识别，对单个字符形状的感知能力较弱；同时，它们对文本长度表现出不敏感性，且在图像细粒度特征检测方面能力有限。因此，这些结果表明，即使是当前最强的大规模多模态模型，也无法在传统文本任务中匹敌领域特定方法，并在更复杂任务中面临更大挑战。最重要的是，本研究所呈现的基线结果可为创新策略的构思与评估提供基础框架，以增强零样本多模态技术。评估管道将发布于https://github.com/Yuliang-Liu/MultimodalOCR。