Effective sentence embeddings that capture semantic nuances and generalize well across diverse contexts are crucial for natural language processing tasks. We address this challenge by applying SimCSE (Simple Contrastive Learning of Sentence Embeddings) using contrastive learning to fine-tune the minBERT model for sentiment analysis, semantic textual similarity (STS), and paraphrase detection. Our contributions include experimenting with three different dropout techniques, namely standard dropout, curriculum dropout, and adaptive dropout, to tackle overfitting, proposing a novel 2-Tier SimCSE Fine-tuning Model that combines both unsupervised and supervised SimCSE on STS task, and exploring transfer learning potential for Paraphrase and SST tasks. Our findings demonstrate the effectiveness of SimCSE, with the 2-Tier model achieving superior performance on the STS task, with an average test score of 0.742 across all three downstream tasks. The results of error analysis reveals challenges in handling complex sentiments and reliance on lexical overlap for paraphrase detection, highlighting areas for future research. The ablation study revealed that removing Adaptive Dropout in the Single-Task Unsupervised SimCSE Model led to improved performance on the STS task, indicating overfitting due to added parameters. Transfer learning from SimCSE models on Paraphrase and SST tasks did not enhance performance, suggesting limited transferability of knowledge from the STS task.

翻译：能够捕捉语义细微差别并在多样化语境中良好泛化的有效句子嵌入对于自然语言处理任务至关重要。我们通过应用SimCSE（句子嵌入的简单对比学习），利用对比学习对minBERT模型进行情感分析、语义文本相似性（STS）和复述检测的微调，以应对这一挑战。我们的贡献包括：实验三种不同的dropout技术（即标准dropout、课程dropout和自适应dropout）以应对过拟合问题；提出一种新颖的双层SimCSE微调模型，该模型在STS任务上结合了无监督和有监督的SimCSE；以及探索复述和SST任务的迁移学习潜力。我们的研究结果证明了SimCSE的有效性，其中双层模型在STS任务上取得了优异性能，在所有三个下游任务中的平均测试得分为0.742。错误分析的结果揭示了处理复杂情感的挑战以及复述检测对词汇重叠的依赖，突出了未来研究的重点方向。消融研究表明，在单任务无监督SimCSE模型中移除自适应dropout可提升STS任务的性能，这表明额外参数导致了过拟合。从SimCSE模型到复述和SST任务的迁移学习并未提升性能，表明STS任务的知识可迁移性有限。