Depth-recurrence facilitates latent reasoning by sharing parameters across depths. However, prior work lacks combined FLOP-, parameter-, and memory-matched baselines, underutilizes depth-recurrence due to partially fixed layer stacks, and ignores the bottleneck of constant hidden-sizes that restricts many-step latent reasoning. To address this, we introduce a modular framework of depth-recurrent attention mixtures (Dreamer), combining sequence attention, depth attention, and sparse expert attention. It alleviates the hidden-size bottleneck through attention along depth, decouples scaling dimensions, and allows depth-recurrent models to scale efficiently and effectively. Across language reasoning benchmarks, our models require 2 to 8x fewer training tokens for the same accuracy as FLOP-, parameter-, and memory-matched SOTA, and outperform ca. 2x larger SOTA models with the same training tokens. We further present insights into knowledge usage across depths, e.g., showing 2 to 11x larger expert selection diversity than SOTA MoEs.

翻译：深度循环通过跨深度共享参数促进潜在推理。然而，先前研究缺乏计算量、参数量和内存匹配的联合基线，由于部分固定的层堆栈而未充分利用深度循环，且忽视了恒定隐藏大小对多步潜在推理的限制瓶颈。为解决这些问题，我们引入了一种模块化的深度循环注意力混合框架（Dreamer），它结合了序列注意力、深度注意力和稀疏专家注意力。该框架通过沿深度方向的注意力缓解了隐藏大小瓶颈，解耦了缩放维度，并使深度循环模型能够高效且有效地扩展。在语言推理基准测试中，我们的模型在达到相同精度时，所需训练词元数量比计算量、参数量和内存匹配的当前最优方法少2至8倍，并在使用相同训练词元时，性能优于约2倍大的当前最优模型。我们进一步揭示了知识在不同深度间的使用机制，例如，展示了比当前最优混合专家模型（MoEs）高2至11倍的专家选择多样性。