The most popular methods for measuring importance of the variables in a black box prediction algorithm make use of synthetic inputs that combine predictor variables from multiple subjects. These inputs can be unlikely, physically impossible, or even logically impossible. As a result, the predictions for such cases can be based on data very unlike any the black box was trained on. We think that users cannot trust an explanation of the decision of a prediction algorithm when the explanation uses such values. Instead we advocate a method called Cohort Shapley that is grounded in economic game theory and unlike most other game theoretic methods, it uses only actually observed data to quantify variable importance. Cohort Shapley works by narrowing the cohort of subjects judged to be similar to a target subject on one or more features. We illustrate it on an algorithmic fairness problem where it is essential to attribute importance to protected variables that the model was not trained on.

翻译：黑箱预测算法中变量重要性的最流行测量方法，往往利用来自多个受试者的预测变量组合生成合成输入。这些输入可能是不可信的、物理上不可能甚至逻辑上不可能的。因此，针对这类案例的预测可能基于与黑箱训练数据截然不同的数据。我们认为，当解释使用此类数值时，用户无法信任预测算法的决策解释。我们转而倡导一种基于经济博弈论的方法——Cohort Shapley。与大多数其他博弈论方法不同，该方法仅使用实际观测数据量化变量重要性。Cohort Shapley通过缩窄在某个或多个特征上与目标受试者相似的受试者群体来实现计算。我们以算法公平性问题为例进行说明——在该问题中，将重要性归因于模型未经训练的受保护变量至关重要。