Traditional comparative learning sentence embedding directly uses the encoder to extract sentence features, and then passes in the comparative loss function for learning. However, this method pays too much attention to the sentence body and ignores the influence of some words in the sentence on the sentence semantics. To this end, we propose CMLM-CSE, an unsupervised contrastive learning framework based on conditional MLM. On the basis of traditional contrastive learning, an additional auxiliary network is added to integrate sentence embedding to perform MLM tasks, forcing sentence embedding to learn more masked word information. Finally, when Bertbase was used as the pretraining language model, we exceeded SimCSE by 0.55 percentage points on average in textual similarity tasks, and when Robertabase was used as the pretraining language model, we exceeded SimCSE by 0.3 percentage points on average in textual similarity tasks.

翻译：传统对比学习句子嵌入方法直接利用编码器提取句子特征，并传入对比损失函数进行学习。然而，该方法过度关注句子主体，忽视了句子中部分词汇对句子语义的影响。为此，我们提出CMLM-CSE，一种基于条件MLM的无监督对比学习框架。在传统对比学习基础上，新增辅助网络整合句子嵌入以执行MLM任务，迫使句子嵌入学习更多掩码词信息。最终，当使用Bertbase作为预训练语言模型时，我们在文本相似度任务上平均超越SimCSE 0.55个百分点；当使用Robertabase作为预训练语言模型时，我们在文本相似度任务上平均超越SimCSE 0.3个百分点。