Spurious correlations in training data significantly hinder the generalization capability of machine learning models when faced with distribution shifts in real-world scenarios. To tackle the problem, numerous debias approaches have been proposed and benchmarked on datasets intentionally designed with severe biases. However, it remains to be asked: \textit{1. Do existing benchmarks really capture biases in the real world? 2. Can existing debias methods handle biases in the real world?} To answer the questions, we revisit biased distributions in existing benchmarks and real-world datasets, and propose a fine-grained framework for analyzing dataset bias by disentangling it into the magnitude and prevalence of bias. We observe and theoretically demonstrate that existing benchmarks poorly represent real-world biases. We further introduce two novel biased distributions to bridge this gap, forming a nuanced evaluation framework for real-world debiasing. Building upon these results, we evaluate existing debias methods with our evaluation framework. Results show that existing methods are incapable of handling real-world biases. Through in-depth analysis, we propose a simple yet effective approach that can be easily applied to existing debias methods, named Debias in Destruction (DiD). Empirical results demonstrate the superiority of DiD, improving the performance of existing methods on all types of biases within the proposed evaluation framework.

翻译：训练数据中的虚假相关性显著阻碍了机器学习模型在面对真实世界场景中分布变化时的泛化能力。为解决此问题，众多去偏方法被提出，并在人为设计具有严重偏差的数据集上进行了基准测试。然而，仍需探讨以下问题：\textit{1. 现有基准测试是否真正捕捉了真实世界中的偏差？ 2. 现有的去偏方法能否处理真实世界中的偏差？} 为回答这些问题，我们重新审视了现有基准测试和真实世界数据集中的有偏分布，并提出一个细粒度框架，通过将数据集偏差解耦为偏差的幅度和普遍性来进行分析。我们观察并从理论上证明，现有基准测试难以代表真实世界的偏差。我们进一步引入了两种新颖的有偏分布来弥合这一差距，形成了一个用于真实世界去偏的细致评估框架。基于这些结果，我们使用我们的评估框架评估了现有的去偏方法。结果表明，现有方法无法处理真实世界的偏差。通过深入分析，我们提出了一种简单而有效的方法，可以轻松应用于现有的去偏方法，命名为“破坏性去偏”（DiD）。实证结果证明了DiD的优越性，它在所提出的评估框架内，针对所有类型的偏差提升了现有方法的性能。