Generative transformer models have become increasingly complex, with large numbers of parameters and the ability to process multiple input modalities. Current methods for explaining their predictions are resource-intensive. Most crucially, they require prohibitively large amounts of extra memory, since they rely on backpropagation which allocates almost twice as much GPU memory as the forward pass. This makes it difficult, if not impossible, to use them in production. We present AtMan that provides explanations of generative transformer models at almost no extra cost. Specifically, AtMan is a modality-agnostic perturbation method that manipulates the attention mechanisms of transformers to produce relevance maps for the input with respect to the output prediction. Instead of using backpropagation, AtMan applies a parallelizable token-based search method based on cosine similarity neighborhood in the embedding space. Our exhaustive experiments on text and image-text benchmarks demonstrate that AtMan outperforms current state-of-the-art gradient-based methods on several metrics while being computationally efficient. As such, AtMan is suitable for use in large model inference deployments.

翻译：生成式Transformer模型已变得日益复杂，拥有大量参数并能处理多种输入模态。当前用于解释其预测的方法在资源消耗上颇为高昂。最关键的是，它们需要极其大量的额外内存，因为依赖反向传播会分配几乎两倍于前向传播的GPU内存。这使得在生产环境中使用它们变得困难甚至不可能。我们提出了AtMan，该方法能以几乎零额外成本提供生成式Transformer模型的解释。具体而言，AtMan是一种模态无关的扰动方法，它通过操控Transformer的注意力机制，生成输入相对于输出预测的相关性图。AtMan不采用反向传播，而是基于嵌入空间中的余弦相似度邻域，应用一种可并行化的逐令牌搜索方法。我们在文本和图文基准上进行的详尽实验表明，AtMan在多项指标上超越了当前最先进的基于梯度的方法，同时保持了计算高效性。因此，AtMan适用于大型模型推理部署场景。