Recently, using large pretrained Transformer models for transfer learning tasks has evolved to the point where they have become one of the flagship trends in the Natural Language Processing (NLP) community, giving rise to various outlooks such as prompt-based, adapters or combinations with unsupervised approaches, among many others. This work proposes a 3 Phase technique to adjust a base model for a classification task. First, we adapt the model's signal to the data distribution by performing further training with a Denoising Autoencoder (DAE). Second, we adjust the representation space of the output to the corresponding classes by clustering through a Contrastive Learning (CL) method. In addition, we introduce a new data augmentation approach for Supervised Contrastive Learning to correct the unbalanced datasets. Third, we apply fine-tuning to delimit the predefined categories. These different phases provide relevant and complementary knowledge to the model to learn the final task. We supply extensive experimental results on several datasets to demonstrate these claims. Moreover, we include an ablation study and compare the proposed method against other ways of combining these techniques.

翻译：近年来，利用大型预训练Transformer模型进行迁移学习任务已发展成为自然语言处理（NLP）领域的旗舰趋势之一，催生了多种技术方向，例如基于提示的学习、适配器方法、与无监督学习方法的结合等。本研究提出了一种三阶段技术，用于针对分类任务调整基础模型。首先，我们通过使用去噪自编码器（DAE）进行进一步训练，使模型的信号适应数据分布。其次，我们通过对比学习（CL）方法进行聚类，将输出表示空间调整至对应类别。此外，我们引入了一种新的监督对比学习数据增强方法，以校正不平衡数据集。第三，我们应用微调来界定预定义类别。这些不同阶段为模型提供了相关且互补的知识，以学习最终任务。我们在多个数据集上提供了广泛的实验结果以验证上述主张。此外，我们还进行了消融研究，并将所提方法与其他结合这些技术的方式进行了比较。