Recently, studies exemplified by Hyper-Connections (HC) have extended the ubiquitous residual connection paradigm established over the past decade by expanding the residual stream width and diversifying connectivity patterns. While yielding substantial performance gains, this diversification fundamentally compromises the identity mapping property intrinsic to the residual connection, which causes severe training instability and restricted scalability, and additionally incurs notable memory access overhead. To address these challenges, we propose Manifold-Constrained Hyper-Connections (mHC), a general framework that projects the residual connection space of HC onto a specific manifold to restore the identity mapping property, while incorporating rigorous infrastructure optimization to ensure efficiency. Empirical experiments demonstrate that mHC is effective for training at scale, offering tangible performance improvements and superior scalability. We anticipate that mHC, as a flexible and practical extension of HC, will contribute to a deeper understanding of topological architecture design and suggest promising directions for the evolution of foundational models.

翻译：近年来，以超连接（HC）为代表的研究通过扩展残差流宽度和多样化连接模式，拓展了过去十年间建立的普遍残差连接范式。虽然这种多样化带来了显著的性能提升，但它从根本上损害了残差连接固有的恒等映射特性，导致严重的训练不稳定性和受限的可扩展性，同时还产生了显著的内存访问开销。为应对这些挑战，我们提出流形约束超连接（mHC）——一种将HC的残差连接空间投影到特定流形以恢复恒等映射特性的通用框架，同时结合严格的基础设施优化以确保效率。实证实验表明，mHC能有效支持大规模训练，提供切实的性能改进和卓越的可扩展性。我们预期mHC作为HC的灵活实用扩展，将有助于深化对拓扑架构设计的理解，并为基础模型的演进指明有前景的方向。