Unsupervised sentence representation learning is one of the fundamental problems in natural language processing with various downstream applications. Recently, contrastive learning has been widely adopted which derives high-quality sentence representations by pulling similar semantics closer and pushing dissimilar ones away. However, these methods fail to capture the fine-grained ranking information among the sentences, where each sentence is only treated as either positive or negative. In many real-world scenarios, one needs to distinguish and rank the sentences based on their similarities to a query sentence, e.g., very relevant, moderate relevant, less relevant, irrelevant, etc. In this paper, we propose a novel approach, RankCSE, for unsupervised sentence representation learning, which incorporates ranking consistency and ranking distillation with contrastive learning into a unified framework. In particular, we learn semantically discriminative sentence representations by simultaneously ensuring ranking consistency between two representations with different dropout masks, and distilling listwise ranking knowledge from the teacher. An extensive set of experiments are conducted on both semantic textual similarity (STS) and transfer (TR) tasks. Experimental results demonstrate the superior performance of our approach over several state-of-the-art baselines.

翻译：无监督句子表示学习是自然语言处理中的基础问题之一，具有多种下游应用场景。近年来，对比学习被广泛采用，通过拉近相似语义、推开不相似语义来获得高质量的句子表示。然而，这些方法未能捕捉句子间的细粒度排序信息，每个句子仅被简单地视为正样本或负样本。在许多实际场景中，需要根据句子与查询句子的相似度对其进行区分和排序，例如非常相关、中等相关、较少相关、不相关等。本文提出了一种名为RankCSE的新型无监督句子表示学习方法，将排序一致性、排序蒸馏与对比学习整合到一个统一框架中。具体而言，我们通过同时确保不同dropout掩码下两个表示之间的排序一致性，并从教师模型中蒸馏列表级排序知识，来学习具有语义判别力的句子表示。我们在语义文本相似度（STS）和迁移（TR）任务上进行了大量实验。实验结果表明，我们的方法在多个最先进基线模型上取得了优越性能。