The degree of semantic relatedness of two units of language has long been considered fundamental to understanding meaning. Additionally, automatically determining relatedness has many applications such as question answering and summarization. However, prior NLP work has largely focused on semantic similarity, a subset of relatedness, because of a lack of relatedness datasets. In this paper, we introduce a dataset for Semantic Textual Relatedness, STR-2022, that has 5,500 English sentence pairs manually annotated using a comparative annotation framework, resulting in fine-grained scores. We show that human intuition regarding relatedness of sentence pairs is highly reliable, with a repeat annotation correlation of 0.84. We use the dataset to explore questions on what makes sentences semantically related. We also show the utility of STR-2022 for evaluating automatic methods of sentence representation and for various downstream NLP tasks. Our dataset, data statement, and annotation questionnaire can be found at: https://doi.org/10.5281/zenodo.7599667

翻译：两个语言单位之间的语义关联程度长期以来被认为是理解意义的基础。此外，自动判定语义关联度在诸多应用中具有重要价值，例如问答系统和文本摘要。然而，由于缺乏关联度数据集，先前的自然语言处理研究主要集中于语义相似性（关联度的一个子集）。本文提出了一个语义文本关联度数据集STR-2022，该数据集包含5,500个英文句子对，采用比较标注框架进行人工标注，生成了细粒度的得分。我们证明，人类对于句对关联度的直觉具有高度可靠性，重复标注相关性达到0.84。利用该数据集，我们探究了句子语义相关的决定因素。同时，我们展示了STR-2022在评估自动句子表示方法及各类下游NLP任务中的实用性。数据集、数据声明及标注问卷可通过以下链接获取：https://doi.org/10.5281/zenodo.7599667