A central idea in mechanistic interpretability is that neural networks represent more features than they have dimensions, arranging them in superposition to form an over-complete basis. This framing has been influential, motivating dictionary learning approaches such as sparse autoencoders. However, superposition has mostly been studied in idealized settings where features are sparse and uncorrelated. In these settings, superposition is typically understood as introducing interference that must be minimized geometrically and filtered out by non-linearities such as ReLUs, yielding local structures like regular polytopes. We show that this account is incomplete for realistic data by introducing Bag-of-Words Superposition (BOWS), a controlled setting to encode binary bag-of-words representations of internet text in superposition. Using BOWS, we find that when features are correlated, interference can be constructive rather than just noise to be filtered out. This is achieved by arranging features according to their co-activation patterns, making interference between active features constructive, while still using ReLUs to avoid false positives. We show that this kind of arrangement is more prevalent in models trained with weight decay and naturally gives rise to semantic clusters and cyclical structures which have been observed in real language models yet were not explained by the standard picture of superposition. Code for this paper can be found at https://github.com/LucasPrietoAl/correlations-feature-geometry.

翻译：机制可解释性中的一个核心观点是，神经网络所表示的特征数量超过其维度数，通过叠加的方式将这些特征排列成过完备基。这一框架具有重要影响，推动了稀疏自编码器等字典学习方法的发展。然而，叠加现象主要在特征稀疏且不相关的理想化场景中被研究。在这些场景中，叠加通常被理解为引入了干扰，需要通过几何方式最小化，并由ReLU等非线性激活函数滤除，从而产生如正多胞体等局部结构。本文通过引入词袋叠加（BOWS）这一受控设置——用于在叠加中编码互联网文本的二进制词袋表示——证明上述解释对于真实数据是不完备的。利用BOWS，我们发现当特征相关时，干扰可以具有建设性作用，而不仅仅是需要滤除的噪声。这是通过根据特征的共激活模式来排列特征实现的，使得活跃特征之间的干扰具有建设性，同时仍利用ReLU避免误报。我们证明，这种排列方式在权重衰减训练的模型中更为普遍，并且自然会产生语义簇和循环结构——这些结构在实际语言模型中已被观察到，但无法用标准的叠加理论解释。本文代码可在 https://github.com/LucasPrietoAl/correlations-feature-geometry 获取。