Large models have recently played a dominant role in natural language processing and multimodal vision-language learning. It remains less explored about their efficacy in text-related visual tasks. We conducted a comprehensive study of existing publicly available multimodal models, evaluating their performance in text recognition (document text, artistic text, handwritten text, scene text), text-based visual question answering (document text, scene text, and bilingual text), key information extraction (receipts, documents, and nutrition facts) and handwritten mathematical expression recognition. Our findings reveal strengths and weaknesses in these models, which primarily rely on semantic understanding for word recognition and exhibit inferior perception of individual character shapes. They also display indifference towards text length and have limited capabilities in detecting finegrained features in images. Consequently, these results demonstrate that even the current most powerful large multimodal models cannot match domain-specific methods in traditional text tasks and face greater challenges in more complex tasks. Most importantly, the baseline results showcased in this study could provide a foundational framework for the conception and assessment of innovative strategies targeted at enhancing zero-shot multimodal techniques. Evaluation pipeline is available at https://github.com/Yuliang-Liu/MultimodalOCR.

翻译：近年来，大语言模型在自然语言处理和视觉-语言多模态学习中占据主导地位，但其在文本相关视觉任务中的效能仍鲜有探讨。我们系统研究了现有公开多模态模型在以下任务中的表现：文本识别（文档文本、艺术文本、手写文本、场景文本）、基于文本的视觉问答（文档文本、场景文本及双语文本）、关键信息提取（收据、文档与营养成分表）以及手写数学表达式识别。研究发现，这些模型主要依赖语义理解进行词汇识别，对单个字符形状的感知能力较弱，存在对文本长度不敏感、难以捕捉图像细粒度特征等局限。结果表明，即使当前最强大的多模态大语言模型，在传统文本任务中仍无法匹敌领域专用方法，且在更复杂任务中面临更大挑战。尤为重要的是，本研究呈现的基线结果可为增强零样本多模态技术的创新策略构思与评估提供基础框架。评估流程详见https://github.com/Yuliang-Liu/MultimodalOCR。