Understanding how Large Language Models (LLMs) process information from prompts remains a significant challenge. To shed light on this "black box," attention visualization techniques have been developed to capture neuron-level perceptions and interpret how models focus on different parts of input data. However, many existing techniques are tailored to specific model architectures, particularly within the Transformer family, and often require backpropagation, resulting in nearly double the GPU memory usage and increased computational cost. A lightweight, model-agnostic approach for attention visualization remains lacking. In this paper, we introduce a model-agnostic token importance visualization technique to better understand how generative AI systems perceive and prioritize information from input text, without incurring additional computational cost. Our method leverages perturbation-based strategies combined with a three-matrix analytical framework to generate relevance maps that illustrate token-level contributions to model predictions. The framework comprises: (1) the Angular Deviation Matrix, which captures shifts in semantic direction; (2) the Magnitude Deviation Matrix, which measures changes in semantic intensity; and (3) the Dimensional Importance Matrix, which evaluates contributions across individual vector dimensions. By systematically removing each token and measuring the resulting impact across these three complementary dimensions, we derive a composite importance score that provides a nuanced and mathematically grounded measure of token significance. To support reproducibility and foster wider adoption, we provide open-source implementations of all proposed and utilized explainability techniques, with code and resources publicly available at https://github.com/Infosys/Infosys-Responsible-AI-Toolkit

翻译：理解大型语言模型（LLMs）如何从提示中处理信息仍是一个重大挑战。为探究这一“黑箱”，注意力可视化技术已被开发用于捕捉神经元层面的感知，并解释模型如何聚焦于输入数据的不同部分。然而，许多现有技术针对特定模型架构（尤其是Transformer家族）量身定制，且往往依赖反向传播，导致GPU内存占用近乎翻倍并增加计算开销。当前仍缺乏一种轻量级、模型无关的注意力可视化方法。本文提出一种模型无关的令牌重要性可视化技术，以更好理解生成式AI系统如何感知和优先处理输入文本中的信息，且无需额外计算成本。该方法结合基于扰动的策略与三矩阵分析框架，生成展示令牌级对模型预测贡献的相关性图。该框架包含：（1）角偏差矩阵——捕获语义方向变化；（2）幅度偏差矩阵——量化语义强度变化；（3）维度重要性矩阵——评估各向量维度的贡献。通过系统性地移除每个令牌并测量其对这三个互补维度的影响，我们推导出复合重要性分数，从而提供精细化且数学严谨的令牌显著性度量。为支持可复现性并促进广泛应用，我们开源了所有提出及使用的可解释性技术实现，代码与资源公开于https://github.com/Infosys/Infosys-Responsible-AI-Toolkit。