Several proposals have been put forward in recent years for improving out-of-distribution (OOD) performance through mitigating dataset biases. A popular workaround is to train a robust model by re-weighting training examples based on a secondary biased model. Here, the underlying assumption is that the biased model resorts to shortcut features. Hence, those training examples that are correctly predicted by the biased model are flagged as being biased and are down-weighted during the training of the main model. However, assessing the importance of an instance merely based on the predictions of the biased model may be too naive. It is possible that the prediction of the main model can be derived from another decision-making process that is distinct from the behavior of the biased model. To circumvent this, we introduce a fine-tuning strategy that incorporates the similarity between the main and biased model attribution scores in a Product of Experts (PoE) loss function to further improve OOD performance. With experiments conducted on natural language inference and fact verification benchmarks, we show that our method improves OOD results while maintaining in-distribution (ID) performance.

翻译：近年来，针对通过减轻数据集偏差来改善分布外（OOD）性能已提出多项方案。一种常见策略是：基于辅助有偏模型对训练样本重新加权，进而训练鲁棒模型。其核心假设在于，有偏模型会依赖于捷径特征。因此，被有偏模型正确预测的训练样本会被标记为有偏样本，并在主模型训练过程中降低其权重。然而，仅依赖有偏模型的预测结果来评估样本重要性可能过于简单——主模型的预测可能源于与有偏模型行为不同的另一种决策过程。为解决此问题，我们提出一种微调策略，通过在主模型与有偏模型的归因分数相似度基础上引入专家乘积（PoE）损失函数，进一步改善OOD性能。在自然语言推理与事实核查基准上的实验表明，本方法在保持分布内（ID）性能的同时提升了OOD结果。