The paradigm of pre-training followed by fine-tuning on downstream tasks has become the mainstream method in natural language processing tasks. Although pre-trained models have the advantage of generalization, their performance may still vary significantly across different domain tasks. This is because the data distribution in different domains varies. For example, the different parts of the sentence 'He married Smt. Dipali Ghosh in 1947 and led a very happy married life' may have different impact for downstream tasks. For similarity calculations, words such as 'led' and 'life' are more important. On the other hand, for sentiment analysis, the word 'happy' is crucial. This indicates that different downstream tasks have different levels of sensitivity to sentence components. Our starting point is to scale information of the model and data according to the specifics of downstream tasks, enhancing domain information of relevant parts for these tasks and reducing irrelevant elements for different domain tasks, called SIFTER. In the experimental part, we use the SIFTER to improve SimCSE by constructing positive sample pairs based on enhancing the sentence stem and reducing the unimportant components in the sentence, and maximize the similarity between three sentences. Similarly, SIFTER can improve the gate mechanism of the LSTM model by short-circuiting the input gate of important words so that the LSTM model remembers the important parts of the sentence. Our experiments demonstrate that SIFTER outperforms the SimCSE and LSTM baselines.

翻译：预训练加下游任务微调的范式已成为自然语言处理任务的主流方法。尽管预训练模型具备泛化优势，但在不同领域任务上的性能仍可能存在显著差异。这是因为不同领域的数据分布存在差异。例如，句子“He married Smt. Dipali Ghosh in 1947 and led a very happy married life”的不同部分对下游任务的影响可能不同。对于相似度计算，“led”和“life”等词更为重要；而对于情感分析，“happy”一词则至关重要。这表明不同下游任务对句子成分的敏感程度存在差异。我们的出发点是根据下游任务的具体特性对模型和数据的特征进行缩放：增强相关部分的领域信息，同时削弱不同领域任务中的无关元素——该方法称为SIFTER。在实验部分，我们采用SIFTER改进SimCSE：通过增强句子主干并弱化次要成分来构建正样本对，最大化三个句子之间的相似度。同样地，SIFTER可通过短路重要词的输入门，使LSTM模型记住句子关键部分，从而改进其门控机制。实验表明，SIFTER在SimCSE和LSTM基线模型上均取得了更优性能。