Machine learning models have been found to learn shortcuts -- unintended decision rules that are unable to generalize -- undermining models' reliability. Previous works address this problem under the tenuous assumption that only a single shortcut exists in the training data. Real-world images are rife with multiple visual cues from background to texture. Key to advancing the reliability of vision systems is understanding whether existing methods can overcome multiple shortcuts or struggle in a Whac-A-Mole game, i.e., where mitigating one shortcut amplifies reliance on others. To address this shortcoming, we propose two benchmarks: 1) UrbanCars, a dataset with precisely controlled spurious cues, and 2) ImageNet-W, an evaluation set based on ImageNet for watermark, a shortcut we discovered affects nearly every modern vision model. Along with texture and background, ImageNet-W allows us to study multiple shortcuts emerging from training on natural images. We find computer vision models, including large foundation models -- regardless of training set, architecture, and supervision -- struggle when multiple shortcuts are present. Even methods explicitly designed to combat shortcuts struggle in a Whac-A-Mole dilemma. To tackle this challenge, we propose Last Layer Ensemble, a simple-yet-effective method to mitigate multiple shortcuts without Whac-A-Mole behavior. Our results surface multi-shortcut mitigation as an overlooked challenge critical to advancing the reliability of vision systems. The datasets and code are released: https://github.com/facebookresearch/Whac-A-Mole.

翻译：机器学习模型被发现会学习“捷径”——即无法泛化的非预期决策规则——这会削弱模型的可靠性。以往研究在训练数据仅存在单一捷径的脆弱假设下解决这一问题。然而，真实图像中充满了从背景到纹理的多种视觉线索。提升视觉系统可靠性的关键在于理解现有方法能否克服多重捷径，抑或陷入“打地鼠”游戏式的困境，即缓解一个捷径会加剧对其他捷径的依赖。为解决这一不足，我们提出了两个基准：1) UrbanCars，一个包含精准控制伪相关线索的数据集；2) ImageNet-W，一个基于ImageNet的评估集，用于研究水印这一影响几乎所有现代视觉模型的捷径。结合纹理与背景，ImageNet-W使我们能够研究自然图像训练过程中产生的多重捷径。我们发现，计算机视觉模型——包括大规模基础模型——无论其训练集、架构和监督方式如何，在面临多重捷径时都会表现挣扎。即便是专门设计用于对抗捷径的方法，在“打地鼠”困境中也难以奏效。为应对这一挑战，我们提出了最后一层集成（Last Layer Ensemble），这是一种简单却有效的方法，可在避免“打地鼠”行为的同时缓解多重捷径。我们的结果表明，多捷径缓解是提升视觉系统可靠性时一个被忽视的关键挑战。数据集与代码已发布：https://github.com/facebookresearch/Whac-A-Mole。