Nowadays, the interpretation of why a machine learning (ML) model makes certain inferences is as crucial as the accuracy of such inferences. Some ML models like the decision tree possess inherent interpretability that can be directly comprehended by humans. Others like artificial neural networks (ANN), however, rely on external methods to uncover the deduction mechanism. SHapley Additive exPlanations (SHAP) is one of such external methods, which requires a background dataset when interpreting ANNs. Generally, a background dataset consists of instances randomly sampled from the training dataset. However, the sampling size and its effect on SHAP remain to be unexplored. In our empirical study on the MIMIC-III dataset, we show that the two core explanations - SHAP values and variable rankings fluctuate when using different background datasets acquired from random sampling, indicating that users cannot unquestioningly trust the one-shot interpretation from SHAP. Luckily, such fluctuation decreases with the increase of the background dataset size. Also, we notice an U-shape in the stability assessment of SHAP variable rankings, demonstrating that SHAP is more reliable in ranking the most and least important variables compared to moderately important ones. Overall, our results suggest that users should take into account how background data affects SHAP results, with improved SHAP stability as the background sample size increases.

翻译：当前，解释机器学习模型为何得出特定推理结论，其重要性已不亚于推理本身的准确性。诸如决策树等模型具备天然可解释性，可直接被人类理解；而人工神经网络等其他模型则需借助外部方法揭示其推理机制。SHapley Additive exPlanations（SHAP）正是此类外部方法之一，在解释人工神经网络时需要依赖背景数据集。通常，背景数据集由从训练数据集中随机采样的实例构成，但采样规模及其对SHAP的影响尚未得到充分探究。通过对MIMIC-III数据集的实证研究，我们发现：当采用不同随机采样获得的背景数据集时，SHAP的两项核心解释——SHAP值与变量排序——会出现波动，表明用户不应无条件信任SHAP的单次解释结果。所幸这种波动会随背景数据集规模的增大而减小。此外，我们观察到SHAP变量排序的稳定性评估呈现U型分布特征，表明SHAP在排序最重要与最不重要变量时的可靠性高于中等重要性变量。总体而言，我们的研究结果表明：用户需要考虑背景数据对SHAP结果的影响，且增大背景样本量可提升SHAP的稳定性。