Mean decrease in impurity (MDI) is a popular feature importance measure for random forests (RFs). We show that the MDI for a feature $X_k$ in each tree in an RF is equivalent to the unnormalized $R^2$ value in a linear regression of the response on the collection of decision stumps that split on $X_k$. We use this interpretation to propose a flexible feature importance framework called MDI+. Specifically, MDI+ generalizes MDI by allowing the analyst to replace the linear regression model and $R^2$ metric with regularized generalized linear models (GLMs) and metrics better suited for the given data structure. Moreover, MDI+ incorporates additional features to mitigate known biases of decision trees against additive or smooth models. We further provide guidance on how practitioners can choose an appropriate GLM and metric based upon the Predictability, Computability, Stability framework for veridical data science. Extensive data-inspired simulations show that MDI+ significantly outperforms popular feature importance measures in identifying signal features. We also apply MDI+ to two real-world case studies on drug response prediction and breast cancer subtype classification. We show that MDI+ extracts well-established predictive genes with significantly greater stability compared to existing feature importance measures. All code and models are released in a full-fledged python package on Github.

翻译：平均杂质减少量（MDI）是随机森林中一种常用的特征重要性度量方法。我们证明了随机森林每棵树中特征$X_k$的MDI等价于在线性回归中以$X_k$为分裂变量的决策桩集合预测响应变量的非标准化$R^2$值。基于这一解释，我们提出了一个名为MDI+的灵活特征重要性框架。具体而言，MDI+通过允许分析者将线性回归模型和$R^2$指标替换为更适合给定数据结构的正则化广义线性模型和指标，从而泛化了MDI。此外，MDI+引入了额外特征来缓解决策树对加性模型或平滑模型的已知偏差。我们进一步基于可预测性、可计算性、稳定性框架为实践者如何选择恰当的广义线性模型和指标提供了指导。广泛的数据驱动模拟表明，MDI+在识别信号特征方面显著优于流行的特征重要性度量方法。我们还将MDI+应用于两个真实案例研究：药物反应预测和乳腺癌亚型分类。结果表明，与现有特征重要性度量方法相比，MDI+能够以显著更高的稳定性提取已获充分验证的预测基因。所有代码和模型均已作为完整的Python包在Github上发布。