Explainable Artificial Intelligence (XAI) aims to improve the transparency of machine learning (ML) pipelines. We systematize the increasingly growing (but fragmented) microcosm of studies that develop and utilize XAI methods for defensive and offensive cybersecurity tasks. We identify 3 cybersecurity stakeholders, i.e., model users, designers, and adversaries, who utilize XAI for 4 distinct objectives within an ML pipeline, namely 1) XAI-enabled user assistance, 2) XAI-enabled model verification, 3) explanation verification & robustness, and 4) offensive use of explanations. Our analysis of the literature indicates that many of the XAI applications are designed with little understanding of how they might be integrated into analyst workflows -- user studies for explanation evaluation are conducted in only 14% of the cases. The security literature sometimes also fails to disentangle the role of the various stakeholders, e.g., by providing explanations to model users and designers while also exposing them to adversaries. Additionally, the role of model designers is particularly minimized in the security literature. To this end, we present an illustrative tutorial for model designers, demonstrating how XAI can help with model verification. We also discuss scenarios where interpretability by design may be a better alternative. The systematization and the tutorial enable us to challenge several assumptions, and present open problems that can help shape the future of XAI research within cybersecurity.

翻译：可解释人工智能（XAI）旨在提升机器学习（ML）流程的透明度。本文系统梳理了日益增长但碎片化的研究领域，这些研究开发并应用XAI方法解决防御性与攻击性网络安全任务。我们识别出三类网络安全利益相关方（模型用户、设计者与攻击者），他们在ML流程中利用XAI实现四个不同目标：1）XAI驱动的用户辅助，2）XAI驱动的模型验证，3）解释性验证与鲁棒性，4）解释的攻击性利用。文献分析表明，多数XAI应用的研发未充分考量其与分析师工作流的整合方式——仅14%的案例开展了面向解释评估的用户研究。安全领域文献有时未能厘清不同利益相关方的角色差异，例如在向模型用户和设计者提供解释的同时，也将解释暴露给攻击者。此外，模型设计者在安全文献中的角色被严重弱化。为此，本文为模型设计者提供说明性教程，展示XAI如何辅助模型验证，并探讨了"设计即可解释"作为替代方案的适用场景。通过系统梳理与教程设计，我们挑战了若干既有假设，提出若干开放性问题，为XAI在网络安全领域的研究发展方向提供指引。