Neural networks trained on biased datasets tend to inadvertently learn spurious correlations, hindering generalization. We formally prove that (1) samples that exhibit spurious correlations lie on a lower rank manifold relative to the ones that do not; and (2) the depth of a network acts as an implicit regularizer on the rank of the attribute subspace that is encoded in its representations. Leveraging these insights, we present DeNetDM, a novel debiasing method that uses network depth modulation as a way of developing robustness to spurious correlations. Using a training paradigm derived from Product of Experts, we create both biased and debiased branches with deep and shallow architectures and then distill knowledge to produce the target debiased model. Our method requires no bias annotations or explicit data augmentation while performing on par with approaches that require either or both. We demonstrate that DeNetDM outperforms existing debiasing techniques on both synthetic and real-world datasets by 5\%. The project page is available at https://vssilpa.github.io/denetdm/.

翻译：在带有偏置的数据集上训练的神经网络往往会无意中学习到虚假相关性，从而阻碍泛化能力。我们形式化证明了：(1) 呈现虚假相关性的样本位于比未呈现虚假相关性样本更低秩的流形上；(2) 网络的深度可作为对其表征中编码的属性子空间秩的隐式正则化器。基于这些发现，我们提出DeNetDM——一种利用网络深度调制来增强对虚假相关性鲁棒性的新型去偏方法。通过采用源自专家乘积法的训练范式，我们分别构建具有深层和浅层架构的偏置分支与去偏分支，继而通过知识蒸馏生成目标去偏模型。该方法无需偏置标注或显式数据增强，其性能却可与需要其中一种或两种条件的现有方法相媲美。实验表明，DeNetDM在合成数据集和真实数据集上均以5%的优势超越现有去偏技术。项目页面详见 https://vssilpa.github.io/denetdm/。