Post-hoc interpretability methods typically attribute a model's behavior to its components, data, or training trajectory in isolation, and are often tied to a particular level of granularity along the local-to-global spectrum. This leads to explanations that lack a unified view and may miss key interactions. We present ExPLAIND, a theoretically grounded, unified framework that integrates model components, data, and training trajectory while supporting explanations across granularities. We generalize recent work on gradient path kernels, reformulating models trained by AdamW as kernel machines. From the resulting kernel feature maps, we derive novel parameter-wise and step-wise influence scores. We empirically validate the resulting decomposition of model behavior in several settings and apply ExPLAIND to two case studies. Our findings on a Transformer exhibiting Grokking support previously proposed learning phases, while refining the final phase as one in which outer layers align around a representation pipeline learned after memorization. For EuroLLM pretraining, ExPLAIND reveals a two-phase dynamic, with the first characterized by outer-layer MLP learning and the second by increased relative influence of intermediate attention layers. These results establish ExPLAIND as a unified framework for interpreting model behavior and training dynamics.

翻译：事后可解释性方法通常孤立地将模型行为归因于其组件、数据或训练轨迹，且常局限于从局部到全局谱系中的特定粒度。这导致解释缺乏统一视角，可能遗漏关键交互。我们提出ExPLAIND，一个理论严谨的统一框架，集成模型组件、数据和训练轨迹，同时支持跨粒度的解释。我们推广了关于梯度路径核的近期工作，将经由AdamW训练的模型重构为核机器。从所得的核特征图中，我们推导出新颖的参数级和步级影响分数。我们在多种设定下实证验证了所得模型行为分解的有效性，并将ExPLAIND应用于两个案例研究。我们对展现Grokking现象的Transformer的发现支持了先前提出的学习阶段划分，同时将最终阶段细化为外层围绕记忆化后习得的表征管线进行对齐的阶段。对于EuroLLM预训练，ExPLAIND揭示了一个两阶段动态：第一阶段以外层MLP学习为特征，第二阶段以中间注意力层相对影响力的增强为特征。这些结果确立了ExPLAIND作为解释模型行为与训练动态的统一框架。