Quantifying variable importance is essential for answering high-stakes questions in fields like genetics, public policy, and medicine. Current methods generally calculate variable importance for a given model trained on a given dataset. However, for a given dataset, there may be many models that explain the target outcome equally well; without accounting for all possible explanations, different researchers may arrive at many conflicting yet equally valid conclusions given the same data. Additionally, even when accounting for all possible explanations for a given dataset, these insights may not generalize because not all good explanations are stable across reasonable data perturbations. We propose a new variable importance framework that quantifies the importance of a variable across the set of all good models and is stable across the data distribution. Our framework is extremely flexible and can be integrated with most existing model classes and global variable importance metrics. We demonstrate through experiments that our framework recovers variable importance rankings for complex simulation setups where other methods fail. Further, we show that our framework accurately estimates the true importance of a variable for the underlying data distribution. We provide theoretical guarantees on the consistency and finite sample error rates for our estimator. Finally, we demonstrate its utility with a real-world case study exploring which genes are important for predicting HIV load in persons with HIV, highlighting an important gene that has not previously been studied in connection with HIV. Code is available here.

翻译：量化变量重要性对于回答遗传学、公共政策及医学等高风险领域的关键问题至关重要。现有方法通常针对给定数据集上训练的特定模型计算变量重要性。然而，对于同一数据集，可能存在多种能同样有效解释目标变量的模型；若未考虑所有可能的解释，不同研究者可能基于相同数据得出相互矛盾但同样合理的结论。此外，即便考虑了给定数据集的所有可能解释，由于并非所有优质模型都能在合理数据扰动下保持稳定，这些结论可能缺乏泛化性。我们提出了一种新的变量重要性框架，该框架能够在所有优质模型集合中量化变量的重要性，并在数据分布层面保持稳定性。该框架具有极高的灵活性，可与大多数现有模型类别及全局变量重要性指标集成。实验表明，在其他方法失效的复杂模拟设置下，我们的框架能够恢复变量重要性排序。进一步，我们证明该框架可准确估计变量在底层数据分布中的真实重要性。我们为估计量的一致性及有限样本误差率提供了理论保证。最后，通过一项真实世界案例研究（探索哪些基因对预测HIV感染者病毒载量至关重要），我们展示了其实用性，并重点发现了一个此前未在HIV研究中被报道的重要基因。代码已公开。