In the field of document understanding, significant advances have been made in the fine-tuning of Multimodal Large Language Models (MLLMs) with instruction-following data. Nevertheless, the potential of text-grounding capability within text-rich scenarios remains underexplored. In this paper, we present a text-grounding document understanding model, termed TGDoc, which addresses this deficiency by enhancing MLLMs with the ability to discern the spatial positioning of text within images. Empirical evidence suggests that text-grounding improves the model's interpretation of textual content, thereby elevating its proficiency in comprehending text-rich images. Specifically, we compile a dataset containing 99K PowerPoint presentations sourced from the internet. We formulate instruction tuning tasks including text detection, recognition, and spotting to facilitate the cohesive alignment between the visual encoder and large language model. Moreover, we curate a collection of text-rich images and prompt the text-only GPT-4 to generate 12K high-quality conversations, featuring textual locations within text-rich scenarios. By integrating text location data into the instructions, TGDoc is adept at discerning text locations during the visual question process. Extensive experiments demonstrate that our method achieves state-of-the-art performance across multiple text-rich benchmarks, validating the effectiveness of our method.

翻译：文档理解领域通过指令微调多模态大语言模型（MLLMs）已取得显著进展，然而在富含文本场景中文本定位能力的潜力仍未充分挖掘。本文提出一种文本定位文档理解模型TGDoc，通过增强MLLMs对图像中文本空间位置的感知能力来弥补这一不足。实验证据表明，文本定位可提升模型对文本内容的解读能力，进而增强其对文本密集型图像的理解水平。我们具体构建了包含99,000份互联网来源PPT演示文稿的数据集，设计包含文本检测、识别与定位的指令微调任务，以促进视觉编码器与大语言模型之间的协同对齐。此外，我们精选了一批文本密集型图像集，通过仅文本输入的GPT-4生成12,000条高质量对话，其中特别标注文本在复杂场景中的空间位置。通过将文本位置数据融入指令，TGDoc可在视觉问答过程中精准定位文本。大量实验表明，本方法在多个文本密集型基准测试中均达到最优性能，验证了该方法的有效性。