The degree of semantic relatedness of two units of language has long been considered fundamental to understanding meaning. Additionally, automatically determining relatedness has many applications such as question answering and summarization. However, prior NLP work has largely focused on semantic similarity, a subset of relatedness, because of a lack of relatedness datasets. In this paper, we introduce a dataset for Semantic Textual Relatedness, STR-2022, that has 5,500 English sentence pairs manually annotated using a comparative annotation framework, resulting in fine-grained scores. We show that human intuition regarding relatedness of sentence pairs is highly reliable, with a repeat annotation correlation of 0.84. We use the dataset to explore questions on what makes sentences semantically related. We also show the utility of STR-2022 for evaluating automatic methods of sentence representation and for various downstream NLP tasks. Our dataset, data statement, and annotation questionnaire can be found at: https://doi.org/10.5281/zenodo.7599667

翻译：两个语言单元的语义相关程度长期以来被视为理解意义的基础。此外，自动确定相关性具有诸多应用，如问答和摘要生成。然而，由于缺乏相关性数据集，先前的自然语言处理研究主要聚焦于语义相似性（相关性的一个子集）。本文中，我们引入了一个语义文本相关性数据集STR-2022，该数据集包含5,500个英文句对，采用比较标注框架进行人工标注，得到了细粒度分数。我们表明，人类对句对相关性的直觉具有高度可靠性，重复标注相关性达到0.84。我们利用该数据集探讨了哪些因素使句子语义相关的问题。同时，我们展示了STR-2022在评估自动句子表征方法及各类下游自然语言处理任务中的效用。我们的数据集、数据声明及标注问卷可通过以下链接获取：https://doi.org/10.5281/zenodo.7599667