The adoption of artificial intelligence (AI) across industries has led to the widespread use of complex black-box models and interpretation tools for decision making. This paper proposes an adversarial framework to uncover the vulnerability of permutation-based interpretation methods for machine learning tasks, with a particular focus on partial dependence (PD) plots. This adversarial framework modifies the original black box model to manipulate its predictions for instances in the extrapolation domain. As a result, it produces deceptive PD plots that can conceal discriminatory behaviors while preserving most of the original model's predictions. This framework can produce multiple fooled PD plots via a single model. By using real-world datasets including an auto insurance claims dataset and COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) dataset, our results show that it is possible to intentionally hide the discriminatory behavior of a predictor and make the black-box model appear neutral through interpretation tools like PD plots while retaining almost all the predictions of the original black-box model. Managerial insights for regulators and practitioners are provided based on the findings.

翻译：在工业界广泛采用人工智能的背景下，复杂黑箱模型及其解释工具在决策过程中得到普遍应用。本文提出一种自适应攻击框架，旨在揭示基于置换的解释方法在机器学习任务中的脆弱性，特别关注偏依赖图（PD图）。该框架通过修改原始黑箱模型，操纵其在插值域实例上的预测结果，从而生成具有欺骗性的PD图，这种图能够掩盖歧视性行为，同时保留原始模型的大多数预测结果。该框架可通过单一模型生成多个被误导的PD图。基于真实世界数据集（包括汽车保险理赔数据集和COMPAS（矫正替代制裁罪犯管理档案）数据集）的实验结果表明，通过PD图等解释工具，能够在保持原始黑箱模型几乎所有预测结果的同时，有意隐藏预测变量的歧视性行为，使黑箱模型看似具有中立性。根据研究发现，本文为监管者和实践者提供了管理启示。