There is an increasing demand for interpretation of model predictions especially in high-risk applications. Various visualization approaches have been proposed to estimate the part of input which is relevant to a specific model prediction. However, most approaches require model structure and parameter details in order to obtain the visualization results, and in general much effort is required to adapt each approach to multiple types of tasks particularly when model backbone and input format change over tasks. In this study, a simple yet effective visualization framework called PAMI is proposed based on the observation that deep learning models often aggregate features from local regions for model predictions. The basic idea is to mask majority of the input and use the corresponding model output as the relative contribution of the preserved input part to the original model prediction. For each input, since only a set of model outputs are collected and aggregated, PAMI does not require any model detail and can be applied to various prediction tasks with different model backbones and input formats. Extensive experiments on multiple tasks confirm the proposed method performs better than existing visualization approaches in more precisely finding class-specific input regions, and when applied to different model backbones and input formats. The source code will be released publicly.

翻译：随着高风险应用场景中对模型预测解释需求的日益增长，多种可视化方法被提出以估计与特定模型预测相关的输入区域。然而，现有方法大多依赖模型结构和参数细节才能获得可视化结果，且当模型主干网络或输入格式随任务变化时，通常需要耗费大量精力适配各类任务。本研究基于深度学习模型常从局部区域聚合特征进行预测这一观察，提出了一种名为PAMI的简洁高效的可视化框架。其核心思想是：通过遮盖大部分输入区域，将对应的模型输出作为保留输入部分对原始模型预测的相对贡献。由于每个输入仅需收集并聚合一组模型输出，PAMI无需任何模型细节，可适用于不同模型主干和输入格式的多种预测任务。在多个任务上的大量实验表明，该方法在精确识别类别特定输入区域方面优于现有可视化方法，且能有效适配不同模型主干和输入格式。相关源代码将公开提供。