We make the information transmitted by attention an explicit, measurable quantity in vision transformers. By inserting variational information bottlenecks on all attention-mediated writes to the residual stream -- without other architectural changes -- we train models with an explicit information cost and obtain a controllable spectrum from independent patch processing to fully expressive global attention. On ImageNet-100, we characterize how classification behavior and information routing evolve across this spectrum, and provide initial insights into how global visual representations emerge from local patch processing by analyzing the first attention heads that transmit information. By biasing learning toward solutions with constrained internal communication, our approach yields models that are more tractable for mechanistic analysis and more amenable to control.

翻译：我们将注意力机制传递的信息在视觉Transformer中显式化、可量化。通过在所有注意力介导的对残差流的写入操作中插入变分信息瓶颈——无需其他架构改动——我们训练出具有显式信息成本的模型，并获得从独立补片处理到完全表达的全局注意力的可控谱系。在ImageNet-100上，我们刻画了分类行为和信息路由如何沿此谱系演化，并通过分析首批传递信息的注意力头，为全局视觉表征如何从局部补片处理中涌现提供了初步见解。通过将学习偏向于内部通信受限的解决方案，我们的方法产生了更易于机制分析且更可控的模型。