In recent years, the field of document understanding has progressed a lot. A significant part of this progress has been possible thanks to the use of language models pretrained on large amounts of documents. However, pretraining corpora used in the domain of document understanding are single domain, monolingual, or nonpublic. Our goal in this paper is to propose an efficient pipeline for creating a big-scale, diverse, multilingual corpus of PDF files from all over the Internet using Common Crawl, as PDF files are the most canonical types of documents as considered in document understanding. We analysed extensively all of the steps of the pipeline and proposed a solution which is a trade-off between data quality and processing time. We also share a CCpdf corpus in a form or an index of PDF files along with a script for downloading them, which produces a collection useful for language model pretraining. The dataset and tools published with this paper offer researchers the opportunity to develop even better multilingual language models.

翻译：近年来，文档理解领域取得了显著进展。这一进展很大程度上得益于在大量文档上预训练的语言模型的应用。然而，当前文档理解领域使用的预训练语料库存在领域单一、语言单一或非公开的问题。本文旨在提出一种高效流程，利用Common Crawl从互联网上大规模构建多样化、多语种的PDF文件语料库——因为PDF文件是文档理解领域中最规范的文档类型。我们深入分析了流程的各个环节，并提出了一种在数据质量与处理时间之间取得平衡的解决方案。同时，我们以PDF文件索引形式共享了CCpdf语料库及配套下载脚本，该集合可用于语言模型预训练。本文发布的数据集与工具为研究者开发更优质的多语种语言模型提供了契机。