Distribution shift is a key challenge for predictive models in practice, creating the need to identify potentially harmful shifts in advance of deployment. Existing work typically defines these worst-case shifts as ones that most degrade the individual-level accuracy of the model. However, when models are used to make a downstream population-level decision like the allocation of a scarce resource, individual-level accuracy may be a poor proxy for performance on the task at hand. We introduce a novel framework that employs a hierarchical model structure to identify worst-case distribution shifts in predictive resource allocation settings by capturing shifts both within and across instances of the decision problem. This task is more difficult than in standard distribution shift settings due to combinatorial interactions, where decisions depend on the joint presence of individuals in the allocation task. We show that the problem can be reformulated as a submodular optimization problem, enabling efficient approximations of worst-case loss. Applying our framework to real data, we find empirical evidence that worst-case shifts identified by one metric often significantly diverge from worst-case distributions identified by other metrics.

翻译：分布偏移是预测模型在实践中面临的关键挑战，这需要在部署前识别潜在的有害偏移。现有研究通常将最坏情况偏移定义为最大程度降低模型个体层面准确性的偏移。然而，当模型被用于做出下游群体层面的决策（如稀缺资源的分配）时，个体层面的准确性可能无法有效反映模型在当前任务上的性能。我们提出了一种新颖的框架，该框架采用分层模型结构，通过捕捉决策问题实例内部及实例之间的偏移，来识别预测性资源分配场景中的最坏情况分布偏移。由于组合交互作用（即决策取决于分配任务中个体的联合存在），该任务比标准分布偏移场景更为复杂。我们证明该问题可被重新表述为一个子模优化问题，从而实现对最坏情况损失的高效近似。将我们的框架应用于实际数据，我们发现了经验证据：由一种度量指标识别的最坏情况偏移，往往与其他度量指标识别的最坏情况分布存在显著差异。