This paper describes Difference-aware Deep continuous prompt for Contrastive Sentence Embeddings (D2CSE) that learns sentence embeddings. Compared to state-of-the-art approaches, D2CSE computes sentence vectors that are exceptional to distinguish a subtle difference in similar sentences by employing a simple neural architecture for continuous prompts. Unlike existing architectures that require multiple pretrained language models (PLMs) to process a pair of the original and corrupted (subtly modified) sentences, D2CSE avoids cumbersome fine-tuning of multiple PLMs by only optimizing continuous prompts by performing multiple tasks -- i.e., contrastive learning and conditional replaced token detection all done in a self-guided manner. D2CSE overloads a single PLM on continuous prompts and greatly saves memory consumption as a result. The number of training parameters in D2CSE is reduced to about 1\% of existing approaches while substantially improving the quality of sentence embeddings. We evaluate D2CSE on seven Semantic Textual Similarity (STS) benchmarks, using three different metrics, namely, Spearman's rank correlation, recall@K for a retrieval task, and the anisotropy of an embedding space measured in alignment and uniformity. Our empirical results suggest that shallow (not too meticulously devised) continuous prompts can be honed effectively for multiple NLP tasks and lead to improvements upon existing state-of-the-art approaches.

翻译：本文提出了一种用于对比句子嵌入的差异感知深度连续提示（D2CSE）方法，以学习句子嵌入。与现有最优方法相比，D2CSE通过采用简单的连续提示神经架构，计算出的句子向量能够卓越地区分相似句子间的细微差异。不同于需要多个预训练语言模型（PLM）处理原始句子与损坏（稍作修改）句子对的现有架构，D2CSE通过执行多项任务——即以自引导方式完成对比学习和条件性替换词检测——仅优化连续提示，从而避免了对多个PLM进行繁琐微调。D2CSE在连续提示上对单个PLM进行过载处理，大幅节省了内存消耗。D2CSE的训练参数数量降至现有方法的约1%，同时显著提升了句子嵌入的质量。我们在七个语义文本相似度（STS）基准上，使用三种不同指标（即斯皮尔曼秩相关系数、检索任务的召回率@K，以及通过对齐性和均匀性测度的嵌入空间各向异性）对D2CSE进行了评估。实证结果表明，浅层（而非精心设计）的连续提示能够有效适应多种自然语言处理任务，并优于现有最优方法。