We study whether the Compressed Computation (CC) toy model (Braun et al., 2025) is an instance of computation in superposition. The CC model appears to compute 100 ReLU functions with just 50 neurons, achieving a better loss than expected from only representing 50 ReLU functions. We show that the model mixes inputs via its noisy residual stream, corresponding to an unintended mixing matrix in the labels. Splitting the training objective into the ReLU term and the mixing term, we find that performance gains scale with the magnitude of the mixing matrix and vanish when the matrix is removed. The learned neuron directions concentrate in the subspace associated with the top 50 eigenvalues of the mixing matrix, suggesting that the mixing term governs the solution. Finally, a semi-non-negative matrix factorization (SNMF) baseline derived solely from the mixing matrix reproduces the qualitative loss profile and improves on prior baselines, though it does not match the trained model. These results suggest CC is not a suitable toy model of computation in superposition.

翻译：我们研究了压缩计算（CC）玩具模型（Braun 等，2025）是否属于叠加计算的实例。该CC模型看似仅用50个神经元就能计算100个ReLU函数，其损失优于仅代表50个ReLU函数时的预期值。我们证明该模型通过其带噪残差流混合输入，对应标签中一个非预期的混合矩阵。将训练目标分解为ReLU项和混合项后，我们发现性能增益随混合矩阵的规模增大而提升，并在移除该矩阵时消失。学习到的神经元方向集中在与混合矩阵前50个特征值相关的子空间中，表明混合项主导了求解过程。最后，仅从混合矩阵导出的半非负矩阵分解（SNMF）基线方法重现了定性损失分布，并优于先前基线，尽管未能完全匹配训练模型。这些结果表明CC并非合适的叠加计算玩具模型。