Quantifying variable importance is essential for answering high-stakes questions in fields like genetics, public policy, and medicine. Current methods generally calculate variable importance for a given model trained on a given dataset. However, for a given dataset, there may be many models that explain the target outcome equally well; without accounting for all possible explanations, different researchers may arrive at many conflicting yet equally valid conclusions given the same data. Additionally, even when accounting for all possible explanations for a given dataset, these insights may not generalize because not all good explanations are stable across reasonable data perturbations. We propose a new variable importance framework that quantifies the importance of a variable across the set of all good models and is stable across the data distribution. Our framework is extremely flexible and can be integrated with most existing model classes and global variable importance metrics. We demonstrate through experiments that our framework recovers variable importance rankings for complex simulation setups where other methods fail. Further, we show that our framework accurately estimates the true importance of a variable for the underlying data distribution. We provide theoretical guarantees on the consistency and finite sample error rates for our estimator. Finally, we demonstrate its utility with a real-world case study exploring which genes are important for predicting HIV load in persons with HIV, highlighting an important gene that has not previously been studied in connection with HIV. Code is available at https://github.com/jdonnelly36/Rashomon_Importance_Distribution.

翻译：量化变量重要性对于遗传学、公共政策和医学等领域中回答高风险问题至关重要。现有方法通常计算给定数据集上训练的特定模型的变量重要性。然而，对于给定的数据集，可能存在多个同样能解释目标结果的模型；若未考虑所有可能的解释，不同研究者基于相同数据可能得出众多矛盾却同样有效的结论。此外，即使考虑了给定数据集的所有可能解释，这些见解也可能无法推广，因为并非所有优秀解释都能在合理的数据扰动下保持稳定。我们提出了一种新的变量重要性框架，该框架能够量化变量在所有优秀模型集合中的重要性，并在数据分布中保持稳定。该框架具有极高的灵活性，可集成至现有的大多数模型类别和全局变量重要性度量中。实验表明，我们的框架能够在其他方法失效的复杂模拟场景中恢复变量重要性排序。进一步地，我们展示了该框架能准确估计变量在底层数据分布中的真实重要性。我们为估计量的一致性及有限样本误差率提供了理论保证。最后，通过真实世界案例研究（探索哪些基因对预测HIV感染者病毒载量重要），我们展示了其应用价值，并揭示了一个此前未被发现与HIV相关的重要基因。代码已发布于 https://github.com/jdonnelly36/Rashomon_Importance_Distribution。