Fine-tuning large pre-trained language models (LLMs) on particular datasets is a commonly employed strategy in Natural Language Processing (NLP) classification tasks. However, this approach usually results in a loss of models generalizability. In this paper, we present a framework that allows for maintaining generalizability, and enhances the performance on the downstream task by utilizing task-specific context attribution. We show that a linear transformation of the text representation from any transformer model using the task-specific concept operator results in a projection onto the latent concept space, referred to as context attribution in this paper. The specific concept operator is optimized during the supervised learning stage via novel loss functions. The proposed framework demonstrates that context attribution of the text representation for each task objective can improve the capacity of the discriminator function and thus achieve better performance for the classification task. Experimental results on three datasets, namely HateXplain, IMDB reviews, and Social Media Attributions, illustrate that the proposed model attains superior accuracy and generalizability. Specifically, for the non-fine-tuned BERT on the HateXplain dataset, we observe 8% improvement in accuracy and 10% improvement in F1-score. Whereas for the IMDB dataset, fine-tuned state-of-the-art XLNet is outperformed by 1% for both accuracy and F1-score. Furthermore, in an out-of-domain cross-dataset test, DistilBERT fine-tuned on the IMDB dataset in conjunction with the proposed model improves the F1-score on the HateXplain dataset by 7%. For the Social Media Attributions dataset of YouTube comments, we observe 5.2% increase in F1-metric. The proposed framework is implemented with PyTorch and provided open-source on GitHub.

翻译：微调特定数据集上的大型预训练语言模型是自然语言处理分类任务中常用的策略。然而，这种方法通常会导致模型泛化能力的丧失。本文提出一个框架，通过利用任务特定的上下文归因，在保持泛化能力的同时提升下游任务性能。我们证明，使用任务特定概念算子对任意Transformer模型的文本表示进行线性变换，可将其投影到潜在概念空间（本文称为上下文归因）。该特定概念算子通过新颖的损失函数在监督学习阶段进行优化。所提框架表明，针对每个任务目标的文本表示上下文归因能够提升判别器函数的容量，从而在分类任务中实现更优性能。在HateXplain、IMDB评论和社交媒体归因三个数据集上的实验结果表明，所提模型取得了优越的准确率和泛化能力。具体而言，对于未微调的BERT在HateXplain数据集上，我们观察到准确率提升8%、F1分数提升10%；而在IMDB数据集上，微调后的先进XLNet模型在准确率和F1分数上均被超越1%。此外，在跨领域跨数据集测试中，基于IMDB数据集微调的DistilBERT结合所提模型，使HateXplain数据集的F1分数提升7%。针对YouTube评论的社交媒体归因数据集，F1指标提升5.2%。该框架采用PyTorch实现，并以开源形式发布在GitHub上。