In the field of document understanding, significant advances have been made in the fine-tuning of Multimodal Large Language Models (MLLMs) with instruction-following data. Nevertheless, the potential of text-grounding capability within text-rich scenarios remains underexplored. In this paper, we present a text-grounding document understanding model, termed TGDoc, which addresses this deficiency by enhancing MLLMs with the ability to discern the spatial positioning of text within images. Empirical evidence suggests that text-grounding improves the model's interpretation of textual content, thereby elevating its proficiency in comprehending text-rich images. Specifically, we compile a dataset containing 99K PowerPoint presentations sourced from the internet. We formulate instruction tuning tasks including text detection, recognition, and spotting to facilitate the cohesive alignment between the visual encoder and large language model. Moreover, we curate a collection of text-rich images and prompt the text-only GPT-4 to generate 12K high-quality conversations, featuring textual locations within text-rich scenarios. By integrating text location data into the instructions, TGDoc is adept at discerning text locations during the visual question process. Extensive experiments demonstrate that our method achieves state-of-the-art performance across multiple text-rich benchmarks, validating the effectiveness of our method.

翻译：在文档理解领域，通过指令微调数据对多模态大语言模型进行微调已取得显著进展。然而，在文本密集场景中，文本定位能力的潜力仍未得到充分探索。本文提出一种文本定位文档理解模型TGDoc，通过增强多模态大语言模型对图像中文本空间位置的感知能力来解决这一不足。实验表明，文本定位提升了模型对文本内容的解读能力，从而增强了其对文本丰富图像的理解水平。具体地，我们构建了一个包含99,000份从互联网获取的PowerPoint演示文稿的数据集，并设计了包含文本检测、识别与定位的指令微调任务，以促进视觉编码器与大语言模型间的协同对齐。此外，我们精心筛选了文本丰富的图像，并引导纯文本GPT-4生成12,000组高质量对话，其中包含文本密集场景下的文本位置信息。通过将文本位置数据融入指令，TGDoc能够在视觉问答过程中准确识别文本位置。大量实验证明，该方法在多个文本密集基准测试中取得了最先进性能，验证了其有效性。