Dense Transformer language models have largely adhered to one consistent architectural shape: each layer consists of an attention module followed by a feed-forward network (FFN) with a narrow-wide-narrow MLP, allocating most parameters to the MLP at expansion ratios between 2 and 4. Motivated by recent results that residual wide-narrow-wide (hourglass) MLPs offer superior function approximation capabilities, we revisit the long-standing MLP shape convention in Transformer, challenging the necessity of the narrow-wide-narrow design. To study this, we develop a Transformer variant that replaces the conventional FFN with a deeper hourglass-shaped FFN, comprising a stack of hourglass sub-MLPs connected by residual pathways. We posit that a deeper but lighter hourglass FFN can serve as a competitive alternative to the conventional FFN, and that parameters saved by using a lighter hourglass FFN can be more effectively utilized, such as by enlarging model hidden dimensions under fixed budgets. We confirm these through empirical validations across model scales: hourglass FFNs outperform conventional FFNs up to 400M and achieve comparable performance at larger scales to 1B parameters; hourglass FFN variants with reduced FFN and increased attention parameters show consistent improvements over conventional configurations at matched budgets. Together, these findings shed new light on recent work and prompt a rethinking of the narrow-wide-narrow MLP convention and the balance between attention and FFN towards efficient and expressive modern language models.

翻译：稠密Transformer语言模型长期以来基本遵循一种一致的架构形态：每一层由一个注意力模块后接一个前馈网络（FFN）构成，该FFN采用窄-宽-窄结构的多层感知机（MLP），并以2至4的扩展比将大部分参数分配给MLP。受近期研究结果的启发——残差连接的宽-窄-宽（沙漏形）MLP展现出更优的函数逼近能力，我们重新审视了Transformer中长期存在的MLP形态惯例，对窄-宽-窄设计的必要性提出质疑。为此，我们开发了一种Transformer变体，将传统的FFN替换为更深的沙漏形FFN，该FFN由一系列通过残差路径连接的沙漏形子MLP堆叠而成。我们提出，一个更深但更轻量的沙漏形FFN可以作为传统FFN的有力替代方案，并且通过使用更轻量的沙漏形FFN所节省的参数可以被更有效地利用，例如在固定参数预算下扩大模型的隐藏维度。我们通过不同模型规模的实证验证证实了这些观点：沙漏形FFN在参数规模达到4亿时优于传统FFN，并在参数规模扩大至10亿时表现出相当的性能；在相同参数预算下，减少FFN参数并增加注意力参数的沙漏形FFN变体相较于传统配置展现出持续的改进。这些发现共同为近期研究提供了新的视角，促使我们重新思考窄-宽-窄MLP惯例以及注意力与FFN之间的平衡，以构建更高效、表达能力更强的现代语言模型。