Visual document understanding is a complex task that involves analyzing both the text and the visual elements in document images. Existing models often rely on manual feature engineering or domain-specific pipelines, which limit their generalization ability across different document types and languages. In this paper, we propose DUBLIN, which is pretrained on webpages using three novel objectives that leverage the spatial and semantic information in the document images: Masked Document Content Generation Task, Bounding Box Task, and Rendered Question Answering Task. We evaluate our model on several benchmarks, such as Web-Based Structural Reading Comprehension, Document Visual Question Answering, Key Information Extraction, Diagram Understanding, and Table Question Answering. We show that our model achieves competitive or better results than the state-of-the-art models on these tasks. In particular, we show that DUBLIN is the first pixel-based model to achieve an EM of 77.75 and F1 of 84.25 on the WebSRC dataset. We also show that our model outperforms the current pixel-based SOTA models on DocVQA and AI2D datasets by significant margins, 2% and 21% increase in performance, respectively. Also, DUBLIN is the first ever pixel-based model which achieves comparable to text-based SOTA methods on XFUND dataset for Semantic Entity Recognition showcasing its multilingual capability. Moreover, we create new baselines for text-based datasets by rendering them as document images and applying this model.

翻译：视觉文档理解是一项复杂任务，需要同时分析文档图像中的文本与视觉元素。现有模型通常依赖人工特征工程或特定领域的流水线设计，这限制了它们在不同文档类型和语言间的泛化能力。本文提出DUBLIN模型，该模型通过三种利用文档图像中空间与语义信息的新型预训练目标在网页数据上进行预训练：掩码文档内容生成任务、边界框任务和渲染式问答任务。我们在多项基准测试上评估模型，包括基于网页的结构化阅读理解、文档视觉问答、关键信息提取、图表理解及表格问答。结果表明，本模型在这些任务中达到与现有最优模型相当或更优的性能。特别地，DUBLIN是首个在WebSRC数据集上达到精确匹配率77.75和F1值84.25的纯像素模型。在DocVQA和AI2D数据集上，我们的模型性能显著超越现有像素级最优模型，分别提升2%和21%。此外，DUBLIN也是首个在XFUND数据集语义实体识别任务中达到与文本级最优方法相当性能的纯像素模型，彰显其多语言处理能力。同时，我们通过将文本型数据集渲染为文档图像并应用本模型，为这些数据集创建了新的基准基线。