Significant developments in techniques such as encoder-decoder models have enabled us to represent information comprising multiple modalities. This information can further enhance many downstream tasks in the field of information retrieval and natural language processing; however, improvements in multi-modal techniques and their performance evaluation require large-scale multi-modal data which offers sufficient diversity. Multi-lingual modeling for a variety of tasks like multi-modal summarization, text generation, and translation leverages information derived from high-quality multi-lingual annotated data. In this work, we present the current largest multi-lingual multi-modal summarization dataset (M3LS), and it consists of over a million instances of document-image pairs along with a professionally annotated multi-modal summary for each pair. It is derived from news articles published by British Broadcasting Corporation(BBC) over a decade and spans 20 languages, targeting diversity across five language roots, it is also the largest summarization dataset for 13 languages and consists of cross-lingual summarization data for 2 languages. We formally define the multi-lingual multi-modal summarization task utilizing our dataset and report baseline scores from various state-of-the-art summarization techniques in a multi-lingual setting. We also compare it with many similar datasets to analyze the uniqueness and difficulty of M3LS.

翻译：编解码器模型等技术的显著发展使我们能够表示包含多种模态的信息。这些信息可以进一步增强信息检索和自然语言处理领域的许多下游任务；然而，多模态技术的改进及其性能评估需要大规模且具备充分多样性的多模态数据。多模态摘要、文本生成和翻译等任务的多语言建模依赖于从高质量多语言标注数据中获取的信息。在本研究中，我们提出了当前规模最大的多语言多模态摘要数据集（M3LS），它包含超过一百万个文档-图像对，以及每对数据经专业标注的多模态摘要。该数据集源自英国广播公司（BBC）十年间发布的新闻文章，涵盖20种语言，针对五种语系实现多样性，同时也是13种语言的最大摘要数据集，并包含2种语言的跨语言摘要数据。我们基于该数据集正式定义了多语言多模态摘要任务，并报告了多种最先进摘要技术在多语言环境下的基线分数。我们还将其与多个类似数据集进行对比，以分析M3LS的独特性和难度。