A diversity of tasks use language models trained on semantic similarity data. While there are a variety of datasets that capture semantic similarity, they are either constructed from modern web data or are relatively small datasets created in the past decade by human annotators. This study utilizes a novel source, newly digitized articles from off-copyright, local U.S. newspapers, to assemble a massive-scale semantic similarity dataset spanning 70 years from 1920 to 1989 and containing nearly 400M positive semantic similarity pairs. Historically, around half of articles in U.S. local newspapers came from newswires like the Associated Press. While local papers reproduced articles from the newswire, they wrote their own headlines, which form abstractive summaries of the associated articles. We associate articles and their headlines by exploiting document layouts and language understanding. We then use deep neural methods to detect which articles are from the same underlying source, in the presence of substantial noise and abridgement. The headlines of reproduced articles form positive semantic similarity pairs. The resulting publicly available HEADLINES dataset is significantly larger than most existing semantic similarity datasets and covers a much longer span of time. It will facilitate the application of contrastively trained semantic similarity models to a variety of tasks, including the study of semantic change across space and time.

翻译：一系列任务依赖于在语义相似度数据上训练的语言模型。尽管存在多种捕捉语义相似度的数据集，但它们要么基于现代网络数据构建，要么是过去十年由人工标注者创建的小规模数据集。本研究利用一种新颖的数据源——美国本地报纸新近数字化的版权过期文章——构建了一个覆盖1920年至1989年共70年、包含近4亿个正例语义相似度对的大规模语义相似度数据集。历史上，美国本地报纸约半数文章来自美联社等新闻通讯社。虽然本地报纸会转载新闻通讯社的文章，但它们会自行撰写标题，这些标题构成了对相关文章的摘要性总结。我们通过利用文档布局和语言理解能力，将文章与其标题关联起来。随后采用深度神经网络方法，在存在大量噪声和删节的情况下，检测哪些文章源自同一原始来源。转载文章的标题构成了正例语义相似度对。最终公开的HEADLINES数据集规模远超现有大多数语义相似度数据集，且覆盖了更长的历史时期。这将促进对比训练语义相似度模型在多种任务中的应用，包括对跨时空语义变化的研究。