Large multimodal models trained on natural documents, which interleave images and text, outperform models trained on image-text pairs on various multimodal benchmarks. However, the datasets used to train these models have not been released, and the collection process has not been fully specified. We introduce the OBELICS dataset, an open web-scale filtered dataset of interleaved image-text documents comprising 141 million web pages extracted from Common Crawl, 353 million associated images, and 115 billion text tokens. We describe the dataset creation process, present comprehensive filtering rules, and provide an analysis of the dataset's content. To show the viability of OBELICS, we train vision and language models of 9 and 80 billion parameters named IDEFICS, and obtain competitive performance on different multimodal benchmarks. We release our dataset, models and code.

翻译：基于自然文档（图像与文本交错排列）训练的大型多模态模型，在各种多模态基准测试中均优于仅使用图像文本配对训练的模型。然而，用于训练这些模型的数据集尚未公开发布，其收集流程也未得到完整说明。本文介绍OBELICS数据集——一个开放的大规模过滤交错图文文档数据集，包含从Common Crawl中提取的1.41亿个网页、3.53亿张关联图像以及1150亿文本标记。我们详述了数据集创建流程，提出了全面的过滤规则，并对数据集内容进行了分析。为验证OBELICS的实用性，我们训练了名为IDEFICS的90亿和800亿参数视觉语言模型，并在多个多模态基准测试中取得具有竞争力的性能。我们公开发布了数据集、模型及代码。