This paper improves contrastive learning for sentence embeddings from two perspectives: handling dropout noise and addressing feature corruption. Specifically, for the first perspective, we identify that the dropout noise from negative pairs affects the model's performance. Therefore, we propose a simple yet effective method to deal with such type of noise. Secondly, we pinpoint the rank bottleneck of current solutions to feature corruption and propose a dimension-wise contrastive learning objective to address this issue. Both proposed methods are generic and can be applied to any contrastive learning based models for sentence embeddings. Experimental results on standard benchmarks demonstrate that combining both proposed methods leads to a gain of 1.8 points compared to the strong baseline SimCSE configured with BERT base. Furthermore, applying the proposed method to DiffCSE, another strong contrastive learning based baseline, results in a gain of 1.4 points.

翻译：本文从两个角度改进了句子嵌入的对比学习：处理Dropout噪声与解决特征损坏问题。具体而言，对于第一个角度，我们识别出负样本对中的Dropout噪声会影响模型性能，因此提出一种简单而有效的方法来处理此类噪声。其次，我们指出现有方案在解决特征损坏时存在秩瓶颈问题，并为此提出一种维度层面的对比学习目标以缓解该缺陷。两种方法均具有通用性，可应用于任何基于对比学习的句子嵌入模型。标准基准实验表明，将两种方法结合使用时，相比以BERT base配置的强基线模型SimCSE，性能提升1.8个百分点。此外，将所提方法应用于另一个强对比学习基线模型DiffCSE，同样获得了1.4个百分点的增益。