We present a novel retrofitting method to induce emotion aspects into pre-trained language models (PLMs) such as BERT and RoBERTa. Our method updates pre-trained network weights using contrastive learning so that the text fragments exhibiting similar emotions are encoded nearby in the representation space, and the fragments with different emotion content are pushed apart. While doing so, it also ensures that the linguistic knowledge already present in PLMs is not inadvertently perturbed. The language models retrofitted by our method, i.e., BERTEmo and RoBERTaEmo, produce emotion-aware text representations, as evaluated through different clustering and retrieval metrics. For the downstream tasks on sentiment analysis and sarcasm detection, they perform better than their pre-trained counterparts (about 1% improvement in F1-score) and other existing approaches. Additionally, a more significant boost in performance is observed for the retrofitted models over pre-trained ones in few-shot learning setting.

翻译：我们提出了一种新颖的改造方法，旨在将情感维度引入预训练语言模型（如BERT和RoBERTa）中。该方法通过对比学习更新预训练网络权重，使具有相似情感内容的文本片段在表示空间中编码位置相近，而情感内容不同的片段则相互远离。在此过程中，该方法同时确保预训练语言模型中已有的语言知识不被意外扰动。经本方法改造的语言模型（即BERTEmo和RoBERTaEmo）能够生成情感感知的文本表示，并通过不同聚类和检索指标进行了评估。在情感分析和讽刺检测的下游任务中，这些模型的性能优于其预训练对应模型（F1分数提升约1%）及其他现有方法。此外，在少样本学习场景中，改造后的模型相比预训练模型表现出更显著的性能提升。