The pre-training for language models captures general language understanding but fails to distinguish the affective impact of a particular context to a specific word. Recent works have sought to introduce contrastive learning (CL) for sentiment-aware pre-training in acquiring affective information. Nevertheless, these methods present two significant limitations. First, the compatibility of the GPU memory often limits the number of negative samples, hindering the opportunities to learn good representations. In addition, using only a few sentiment polarities as hard labels, e.g., positive, neutral, and negative, to supervise CL will force all representations to converge to a few points, leading to the issue of latent space collapse. This study proposes a soft momentum contrastive learning (SoftMCL) for fine-grained sentiment-aware pre-training. Instead of hard labels, we introduce valence ratings as soft-label supervision for CL to fine-grained measure the sentiment similarities between samples. The proposed SoftMCL is conducted on both the word- and sentence-level to enhance the model's ability to learn affective information. A momentum queue was introduced to expand the contrastive samples, allowing storing and involving more negatives to overcome the limitations of hardware platforms. Extensive experiments were conducted on four different sentiment-related tasks, which demonstrates the effectiveness of the proposed SoftMCL method. The code and data of the proposed SoftMCL is available at: https://www.github.com/wangjin0818/SoftMCL/.

翻译：语言模型的预训练虽然能够捕捉通用语言理解，但难以区分特定语境对某个词语的情感影响。近期研究尝试引入对比学习进行情感感知预训练以获取情感信息，然而这些方法存在两个显著局限：其一，GPU内存兼容性往往限制负样本数量，阻碍优质表征的学习机会；其二，仅采用少量情感极性（如积极、中性、消极）作为硬标签监督对比学习，会迫使所有表征收敛到少数几个点，导致潜在空间坍塌问题。本研究提出软动量对比学习（SoftMCL），用于细粒度情感感知预训练。区别于硬标签，我们引入效价评分作为软标签监督对比学习，以细粒度衡量样本间的情感相似性。该SoftMCL方法在词语级和句子级分别实施，以增强模型学习情感信息的能力。通过引入动量队列扩展对比样本，该方法可存储并调用更多负样本以突破硬件平台限制。在四项不同情感相关任务上的大量实验证明了所提SoftMCL方法的有效性，相关代码与数据已开源至：https://www.github.com/wangjin0818/SoftMCL/。