Recently, leveraging large language models (LLMs) or multimodal large language models (MLLMs) for document understanding has been proven very promising. However, previous works that employ LLMs/MLLMs for document understanding have not fully explored and utilized the document layout information, which is vital for precise document understanding. In this paper, we propose LayoutLLM, an LLM/MLLM based method for document understanding. The core of LayoutLLM is a layout instruction tuning strategy, which is specially designed to enhance the comprehension and utilization of document layouts. The proposed layout instruction tuning strategy consists of two components: Layout-aware Pre-training and Layout-aware Supervised Fine-tuning. To capture the characteristics of document layout in Layout-aware Pre-training, three groups of pre-training tasks, corresponding to document-level, region-level and segment-level information, are introduced. Furthermore, a novel module called layout chain-of-thought (LayoutCoT) is devised to enable LayoutLLM to focus on regions relevant to the question and generate accurate answers. LayoutCoT is effective for boosting the performance of document understanding. Meanwhile, it brings a certain degree of interpretability, which could facilitate manual inspection and correction. Experiments on standard benchmarks show that the proposed LayoutLLM significantly outperforms existing methods that adopt open-source 7B LLMs/MLLMs for document understanding. The training data of the LayoutLLM is publicly available at https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/DocumentUnderstanding/LayoutLLM

翻译：近期，利用大语言模型（LLM）或多模态大语言模型（MLLM）进行文档理解已展现出巨大潜力。然而，现有基于LLM/MLLM的文档理解方法尚未充分探索和利用文档布局信息，而布局信息对精确的文档理解至关重要。本文提出LayoutLLM——一种基于LLM/MLLM的文档理解方法。其核心是专门设计的布局指令微调策略，旨在增强模型对文档布局的理解与利用能力。该策略包含两部分：布局感知预训练和布局感知有监督微调。在布局感知预训练阶段，为捕获文档布局特征，引入了三组预训练任务，分别对应文档级、区域级和片段级信息。此外，我们设计了新颖的布局思维链（LayoutCoT）模块，使LayoutLLM能聚焦与问题相关的区域并生成准确答案。LayoutCoT不仅能有效提升文档理解性能，还具备一定可解释性，便于人工检查与修正。标准基准实验表明，所提出的LayoutLLM显著优于现有采用开源7B LLM/MLLM的文档理解方法。LayoutLLM的训练数据已在https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/DocumentUnderstanding/LayoutLLM 公开。