Recent advances in training large language models (LLMs) using massive amounts of solely textual data lead to strong generalization across many domains and tasks, including document-specific tasks. Opposed to that there is a trend to train multi-modal transformer architectures tailored for document understanding that are designed specifically to fuse textual inputs with the corresponding document layout. This involves a separate fine-tuning step for which additional training data is required. At present, no document transformers with comparable generalization to LLMs are available That raises the question which type of model is to be preferred for document understanding tasks. In this paper we investigate the possibility to use purely text-based LLMs for document-specific tasks by using layout enrichment. We explore drop-in modifications and rule-based methods to enrich purely textual LLM prompts with layout information. In our experiments we investigate the effects on the commercial ChatGPT model and the open-source LLM Solar. We demonstrate that using our approach both LLMs show improved performance on various standard document benchmarks. In addition, we study the impact of noisy OCR and layout errors, as well as the limitations of LLMs when it comes to utilizing document layout. Our results indicate that layout enrichment can improve the performance of purely text-based LLMs for document understanding by up to 15% compared to just using plain document text. In conclusion, this approach should be considered for the best model choice between text-based LLM or multi-modal document transformers.

翻译：近期通过海量纯文本数据训练大型语言模型（LLM）取得的进展，使其在包括文档特定任务在内的众多领域与任务中展现出强大的泛化能力。与之相对，为文档理解任务设计的多模态Transformer架构则通过显式融合文本输入与对应文档布局，需要额外训练数据进行独立微调。当前尚不存在泛化能力可与LLM媲美的文档Transformer，这引发了关于何种模型更适用于文档理解任务的讨论。本文探索了通过布局增强技术，使纯文本LLM适用于文档特定任务的可能性。我们研究了即插即用式修改与基于规则的方法，将布局信息注入纯文本LLM的提示中。实验中，我们评估了该方法对商业模型ChatGPT与开源模型Solar的影响，证明两种LLM在多个标准文档基准测试中性能均有提升。此外，我们分析了噪声OCR与布局错误的干扰，以及LLM在利用文档布局方面的局限性。结果表明，相较于纯文本输入，布局增强可使基于文本的LLM在文档理解任务中性能提升最高达15%。综上，在纯文本LLM与多模态文档Transformer的模型选择中，本方法应作为重要考量因素。