In recent years, the use of multi-modal pre-trained Transformers has led to significant advancements in visually-rich document understanding. However, existing models have mainly focused on features such as text and vision while neglecting the importance of layout relationship between text nodes. In this paper, we propose GraphLayoutLM, a novel document understanding model that leverages the modeling of layout structure graph to inject document layout knowledge into the model. GraphLayoutLM utilizes a graph reordering algorithm to adjust the text sequence based on the graph structure. Additionally, our model uses a layout-aware multi-head self-attention layer to learn document layout knowledge. The proposed model enables the understanding of the spatial arrangement of text elements, improving document comprehension. We evaluate our model on various benchmarks, including FUNSD, XFUND and CORD, and achieve state-of-the-art results among these datasets. Our experimental results demonstrate that our proposed method provides a significant improvement over existing approaches and showcases the importance of incorporating layout information into document understanding models. We also conduct an ablation study to investigate the contribution of each component of our model. The results show that both the graph reordering algorithm and the layout-aware multi-head self-attention layer play a crucial role in achieving the best performance.

翻译：近年来，多模态预训练Transformer的使用在视觉丰富文档理解领域取得了显著进展。然而，现有模型主要关注文本和视觉等特征，忽视了文本节点间布局关系的重要性。本文提出GraphLayoutLM，一种新型文档理解模型，通过布局结构图建模将文档布局知识注入模型。GraphLayoutLM利用图重排序算法根据图结构调整文本序列。此外，我们的模型采用布局感知多头自注意力层来学习文档布局知识。该模型能够理解文本元素的空间排列，从而提升文档理解能力。我们在多个基准数据集（包括FUNSD、XFUND和CORD）上评估模型，取得了这些数据集上的最优结果。实验结果表明，所提方法相比现有方法有显著提升，展示了将布局信息融入文档理解模型的重要性。我们还通过消融研究探究各模型组件的贡献，结果证明图重排序算法和布局感知多头自注意力层在实现最佳性能中均发挥了关键作用。