Text-rich VQA, namely Visual Question Answering based on text recognition in the images, is a cross-modal task that requires both image comprehension and text recognition. In this work, we focus on investigating the advantages and bottlenecks of LLM-based approaches in addressing this problem. To address the above concern, we separate the vision and language modules, where we leverage external OCR models to recognize texts in the image and Large Language Models (LLMs) to answer the question given texts. The whole framework is training-free benefiting from the in-context ability of LLMs. This pipeline achieved superior performance compared to the majority of existing Multimodal Large Language Models (MLLM) on four text-rich VQA datasets. Besides, based on the ablation study, we find that LLM brings stronger comprehension ability and may introduce helpful knowledge for the VQA problem. The bottleneck for LLM to address text-rich VQA problems may primarily lie in visual part. We also combine the OCR module with MLLMs and pleasantly find that the combination of OCR module with MLLM also works. It's worth noting that not all MLLMs can comprehend the OCR information, which provides insights into how to train an MLLM that preserves the abilities of LLM.

翻译：文本密集型VQA（基于图像中文本识别的视觉问答）是一项需要同时理解图像和识别文本的跨模态任务。本研究聚焦于探究基于大语言模型（LLM）的方法在处理该问题时的优势与瓶颈。为解答上述问题，我们将视觉与语言模块分离，利用外部OCR模型识别图像中的文本，并借助大语言模型（LLM）根据文本回答问题。得益于LLM的上下文学习能力，整个框架无需训练即可运行。在四个文本密集型VQA数据集上，该流程的性能优于大多数现有的多模态大语言模型（MLLM）。此外，基于消融实验发现：LLM能带来更强的理解能力，并可能为VQA问题引入有用知识；而LLM处理文本密集型VQA问题的瓶颈主要在于视觉部分。我们还尝试将OCR模块与MLLM结合，惊喜地发现两者协同有效。值得注意的是，并非所有MLLM都能理解OCR信息，这为训练保留LLM能力的MLLM提供了重要启示。