Superposition, the ability of neural networks to represent more features than neurons, is increasingly seen as key to the efficiency of large models. This paper investigates the theoretical foundations of computing in superposition, establishing complexity bounds for explicit, provably correct algorithms. We present the first lower bounds for a neural network computing in superposition, showing that for a broad class of problems, including permutations and pairwise logical operations, computing $m'$ features in superposition requires at least $Ω(\sqrt{m' \log m'})$ neurons and $Ω(m' \log m')$ parameters. This implies an explicit limit on how much one can sparsify or distill a model while preserving its expressibility, and complements empirical scaling laws by implying the first subexponential bound on capacity: a network with $n$ neurons can compute at most $O(n^2 / \log n)$ features. Conversely, we provide a nearly tight constructive upper bound: logical operations like pairwise AND can be computed using $O(\sqrt{m'} \log m')$ neurons and $O(m' \log^2 m')$ parameters. There is thus an exponential gap between the complexity of computing in superposition (the subject of this work) versus merely representing features, which can require as little as $O(\log m')$ neurons based on the Johnson-Lindenstrauss Lemma. Our work analytically establishes that the number of parameters is a good estimator of the number of features a neural network computes.

翻译：叠加态——神经网络能够表示比神经元数量更多特征的能力——日益被视为大型模型高效性的关键。本文研究叠加态计算的理论基础，为显式可证明正确算法建立复杂度界限。我们首次提出了神经网络在叠加态计算中的下界，证明对于包括排列和成对逻辑运算在内的广泛问题类别，在叠加态中计算$m'$个特征至少需要$Ω(\sqrt{m' \log m'})$个神经元和$Ω(m' \log m')$个参数。这意味着在保持模型表达能力的前提下，模型稀疏化或蒸馏存在明确限制，并通过首次提出容量的次指数界限补充了经验缩放定律：具有$n$个神经元的网络最多能计算$O(n^2 / \log n)$个特征。反之，我们给出了近乎紧致的构造性上界：诸如成对AND的逻辑运算可使用$O(\sqrt{m'} \log m')$个神经元和$O(m' \log^2 m')$个参数实现。因此，叠加态计算复杂度（本文研究主题）与仅表示特征之间存在指数级差距——基于Johnson-Lindenstrauss引理，后者可能仅需$O(\log m')$个神经元。我们的研究通过理论分析证实，参数数量是衡量神经网络计算特征数量的有效估计指标。