Visual information extraction (VIE) plays an important role in Document Intelligence. Generally, it is divided into two tasks: semantic entity recognition (SER) and relation extraction (RE). Recently, pre-trained models for documents have achieved substantial progress in VIE, particularly in SER. However, most of the existing models learn the geometric representation in an implicit way, which has been found insufficient for the RE task since geometric information is especially crucial for RE. Moreover, we reveal another factor that limits the performance of RE lies in the objective gap between the pre-training phase and the fine-tuning phase for RE. To tackle these issues, we propose in this paper a multi-modal framework, named GeoLayoutLM, for VIE. GeoLayoutLM explicitly models the geometric relations in pre-training, which we call geometric pre-training. Geometric pre-training is achieved by three specially designed geometry-related pre-training tasks. Additionally, novel relation heads, which are pre-trained by the geometric pre-training tasks and fine-tuned for RE, are elaborately designed to enrich and enhance the feature representation. According to extensive experiments on standard VIE benchmarks, GeoLayoutLM achieves highly competitive scores in the SER task and significantly outperforms the previous state-of-the-arts for RE (\eg, the F1 score of RE on FUNSD is boosted from 80.35\% to 89.45\%). The code and models are publicly available at https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/DocumentUnderstanding/GeoLayoutLM

翻译：视觉信息抽取（VIE）在文档智能中扮演着重要角色。通常，VIE分为两个子任务：语义实体识别（SER）和关系抽取（RE）。近年来，基于文档的预训练模型在VIE领域取得了显著进展，尤其在SER任务上表现突出。然而，现有大多数模型以隐式方式学习几何表示，这已被发现不足以支持RE任务，因为几何信息对RE尤为关键。此外，我们揭示了限制RE性能的另一因素：预训练阶段与RE微调阶段之间存在目标差异。为解决这些问题，本文提出一个名为GeoLayoutLM的多模态框架用于VIE。GeoLayoutLM在预训练中显式建模几何关系，我们称之为几何预训练。几何预训练通过三个专门设计的几何相关预训练任务实现。此外，我们精心设计了新型关系头，这些关系头通过几何预训练任务进行预训练，并针对RE进行微调，以丰富和增强特征表示。在标准VIE基准上的大量实验表明，GeoLayoutLM在SER任务上取得了极具竞争力的分数，并在RE任务上显著优于此前的最优方法（例如，FUNSD数据集上RE的F1分数从80.35%提升至89.45%）。代码和模型已在https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/DocumentUnderstanding/GeoLayoutLM 公开提供。