In recent years, the field of document understanding has progressed a lot. A significant part of this progress has been possible thanks to the use of language models pretrained on large amounts of documents. However, pretraining corpora used in the domain of document understanding are single domain, monolingual, or nonpublic. Our goal in this paper is to propose an efficient pipeline for creating a big-scale, diverse, multilingual corpus of PDF files from all over the Internet using Common Crawl, as PDF files are the most canonical types of documents as considered in document understanding. We analysed extensively all of the steps of the pipeline and proposed a solution which is a trade-off between data quality and processing time. We also share a CCpdf corpus in a form or an index of PDF files along with a script for downloading them, which produces a collection useful for language model pretraining. The dataset and tools published with this paper offer researchers the opportunity to develop even better multilingual language models.

翻译：近年来，文档理解领域取得了显著进展。这一进展很大程度上归功于在大量文档上预训练的语言模型的应用。然而，当前文档理解领域使用的预训练语料库存在领域单一、语言单一或非公开的问题。本文的目标是提出一种高效的流水线方法，利用Common Crawl从互联网上构建大规模、多样化、多语种的PDF文件语料库——鉴于PDF文件是文档理解领域中最规范的文档类型。我们对流水线的所有步骤进行了深入分析，并提出了一种在数据质量与处理时间之间取得平衡的解决方案。同时，我们以PDF文件索引形式共享CCpdf语料库，并附带下载脚本，所得数据集可用于语言模型预训练。本文发布的数据集和工具将为研究者开发更优质的多语种语言模型提供支持。