Existing sentence textual similarity benchmark datasets only use a single number to summarize how similar the sentence encoder's decision is to humans'. However, it is unclear what kind of sentence pairs a sentence encoder (SE) would consider similar. Moreover, existing SE benchmarks mainly consider sentence pairs with low lexical overlap, so it is unclear how the SEs behave when two sentences have high lexical overlap. We introduce a high-quality SE diagnostic dataset, HEROS. HEROS is constructed by transforming an original sentence into a new sentence based on certain rules to form a \textit{minimal pair}, and the minimal pair has high lexical overlaps. The rules include replacing a word with a synonym, an antonym, a typo, a random word, and converting the original sentence into its negation. Different rules yield different subsets of HEROS. By systematically comparing the performance of over 60 supervised and unsupervised SEs on HEROS, we reveal that most unsupervised sentence encoders are insensitive to negation. We find the datasets used to train the SE are the main determinants of what kind of sentence pairs an SE considers similar. We also show that even if two SEs have similar performance on STS benchmarks, they can have very different behavior on HEROS. Our result reveals the blind spot of traditional STS benchmarks when evaluating SEs.

翻译：现有句子文本相似度基准数据集仅用单一数值概括句子编码器与人类判断的相似程度，但无法明确何种句子对会被句子编码器判定为相似。此外，现有句子编码器基准主要考虑词汇重叠度较低的句子对，因此当两个句子具有高词汇重叠时，句子编码器的行为尚不明确。我们构建了高质量句子编码器诊断数据集HEROS。HEROS通过将原始句子按照特定规则转换为新句子形成最小配对，且该配对具有高词汇重叠。规则包括用同义词、反义词、拼写错误词、随机词替换，以及将原始句子转换为否定形式。不同规则对应HEROS的不同子集。通过系统比较60余种有监督与无监督句子编码器在HEROS上的性能，我们发现大多数无监督编码器对否定结构不敏感。研究证实，训练句子编码器的数据集是决定其判定句子对相似性的主要因素。我们还发现，即便两个句子编码器在STS基准上表现相似，它们在HEROS上的行为也可能截然不同。该结果揭示了传统STS基准在评估句子编码器时存在的盲区。