Sparse autoencoders (SAEs) are widely used to extract interpretable features from neural network representations, often under the implicit assumption that concepts correspond to independent linear directions. However, a growing body of evidence suggests that many concepts are instead organized along low-dimensional manifolds encoding continuous geometric relationships. This raises three basic questions: what does it mean for an SAE to capture a manifold, when do existing SAE architectures do so, and how? We develop a theoretical framework that answers these questions and show that SAEs can capture manifolds in two fundamentally different ways: globally, by allocating a compact group of atoms whose linear span contains the entire manifold, or locally, by distributing it across features that each selectively tile a restricted region of the underlying geometry. Empirically, we find that SAEs suboptimally recover continuous structures, mixing the global subspace and local tiling solutions in a fragmented regime we call dilution. This explains why manifold structure is rarely visible at the level of individual concepts and motivates post-hoc unsupervised discovery methods that search for coherent groups of atoms rather than isolated directions. More broadly, our results suggest that future representation learning methods should treat geometric objects, not just individual directions, as the basic units of interpretability.

翻译：稀疏自编码器（SAEs）被广泛用于从神经网络表征中提取可解释特征，其隐含假设通常是概念对应于独立的线性方向。然而，越来越多的证据表明，许多概念实际上沿着编码连续几何关系的低维流形组织。这引出了三个基本问题：SAE捕捉流形意味着什么，现有SAE架构在何时以及如何做到这一点？我们提出一个理论框架来回答这些问题，并证明SAE可以通过两种根本不同的方式捕捉流形：全局方式，通过分配一组紧凑的原语，其线性张成空间包含整个流形；或局部方式，通过将流形分布到特征上，每个特征选择性拼贴底层几何中的受限区域。实验上，我们发现SAE次优地恢复连续结构，将全局子空间和局部拼贴解决方案混合在一种我们称为“稀释”的碎片化状态中。这解释了为什么流形结构很少在单个概念层面可见，并激励了后验的无监督发现方法，这些方法搜索原语的连贯组而非孤立方向。更广泛地说，我们的结果表明，未来的表征学习方法应将几何对象（而不仅仅是单个方向）视为可解释性的基本单元。