Deep learning models, particularly Transformers, are often criticized as "black boxes" and lack interpretability. We propose Prism, a white-box attention-based architecture derived from the principles of Maximizing Coding Rate Reduction ($\text{MCR}^2$). By modeling the attention mechanism as a gradient ascent process on a distinct signal-noise manifold, we introduce two physical constraints: an overcomplete dictionary to expand the representational phase space, and an irrational frequency separation ($π$-RoPE) to enforce incoherence between signal and noise subspaces. We demonstrate that these geometric inductive biases can be viewed as a physical constraint and they are sufficient to induce unsupervised functional disentanglement alone. Using TinyStories as a controlled testbed for verifying spectral dynamics, we observe that Prism spontaneously specializes its attention heads into spectrally distinct regimes: low-frequency heads capturing long-range causal dependencies (signal) and high-frequency heads handling local syntactic constraints (noise). Our results suggest that interpretability and performance are not a trade-off, but can be unified through principled geometric construction.

翻译：深度学习模型，尤其是Transformer，常被批评为“黑箱”且缺乏可解释性。我们提出Prism，一种基于注意力机制的白盒架构，其推导源于最大化编码率缩减（$\text{MCR}^2$）原理。通过将注意力机制建模为在特定信号-噪声流形上的梯度上升过程，我们引入了两个物理约束：一个用于扩展表示相空间的过完备字典，以及一个用于强制信号与噪声子空间不相干的非有理频率分离（$\pi$-RoPE）。我们证明这些几何归纳偏置可被视为一种物理约束，并且它们本身足以诱导无监督的功能解耦。使用TinyStories作为验证谱动力学的受控测试平台，我们观察到Prism自发地将其注意力头特化为谱域不同的机制：低频头捕获长程因果依赖（信号），而高频头处理局部句法约束（噪声）。我们的结果表明，可解释性与性能并非权衡关系，而是可以通过原则性的几何构造统一起来。