Interpretability, trustworthiness, and usability are key considerations in high-stake security applications, especially when utilizing deep learning models. While these models are known for their high accuracy, they behave as black boxes in which identifying important features and factors that led to a classification or a prediction is difficult. This can lead to uncertainty and distrust, especially when an incorrect prediction results in severe consequences. Thus, explanation methods aim to provide insights into the inner working of deep learning models. However, most explanation methods provide inconsistent explanations, have low fidelity, and are susceptible to adversarial manipulation, which can reduce model trustworthiness. This paper provides a comprehensive analysis of explainable methods and demonstrates their efficacy in three distinct security applications: anomaly detection using system logs, malware prediction, and detection of adversarial images. Our quantitative and qualitative analysis reveals serious limitations and concerns in state-of-the-art explanation methods in all three applications. We show that explanation methods for security applications necessitate distinct characteristics, such as stability, fidelity, robustness, and usability, among others, which we outline as the prerequisites for trustworthy explanation methods.

翻译：可解释性、可信赖性与可用性是高风险安全应用中的关键考量因素，尤其在利用深度学习模型时更是如此。尽管这些模型以高精度著称，但它们作为黑箱运行，难以识别导致分类或预测结果的重要特征与因素。当错误预测导致严重后果时，这可能引发不确定性与不信任。因此，解释方法旨在揭示深度学习模型的内部工作机制。然而，大多数解释方法提供的解释不一致、保真度低，且易受对抗性操纵影响，从而削弱模型的可信赖性。本文对可解释方法进行了全面分析，并在三种不同安全应用场景中验证了其有效性：基于系统日志的异常检测、恶意软件预测及对抗性图像检测。我们的定量与定性分析揭示了当前最先进的解释方法在所有三种应用场景中均存在严重局限性及问题。研究表明，安全应用场景中的解释方法需要具备稳定性、保真度、鲁棒性与可用性等独特特征，我们将其归纳为可信赖解释方法的前提条件。