Pre-Layer Normalization (Pre-LN) is the de facto choice for large language models (LLMs) and is crucial for stable pretraining and effective transfer learning. However, Pre-LN is inefficient due to repeated statistical calculations and suffers from the curse of depth. As layers grow, the magnitude and variance of the hidden state escalate, destabilizing training. Efficiency-oriented normalization-free methods such as Dynamic Tanh (DyT) improve speed but remain fragile at depth. To jointly address stability and efficiency, we propose Bounded Hyperbolic Tanh (BHyT), a drop-in replacement for Pre-LN. BHyT couples a tanh nonlinearity with explicit, data-driven input bounding to keep activations within a non-saturating range. It prevents depth-wise growth in activation magnitude and variance and comes with a theoretical stability guarantee. For efficiency, BHyT computes exact statistics once per block and replaces a second normalization with a lightweight variance approximation, enhancing efficiency. Empirically, BHyT demonstrates improved stability and efficiency during pretraining, achieving an average of 15.8% faster training and an average of 4.2% higher token generation throughput compared to RMSNorm., while matching or surpassing its inference performance and robustness across language understanding and reasoning benchmarks. Our code is available at: https://anonymous.4open.science/r/BHyT

翻译：预层归一化（Pre-LN）是大型语言模型（LLM）事实上的标准选择，对于稳定预训练和有效迁移学习至关重要。然而，由于重复的统计计算，Pre-LN 效率低下，并遭受深度诅咒的影响。随着层数增加，隐藏状态的幅值和方差随之增大，导致训练不稳定。面向效率的无归一化方法（如动态 Tanh（DyT））提升了速度，但在深度上仍显脆弱。为了同时解决稳定性和效率问题，我们提出了有界双曲正切（BHyT），作为 Pre-LN 的直接替代方案。BHyT 将 tanh 非线性与显式的、数据驱动的输入边界约束相结合，将激活值保持在非饱和范围内。它防止了激活幅值和方差随深度增长，并具有理论上的稳定性保证。在效率方面，BHyT 每个模块仅计算一次精确统计量，并用轻量级的方差近似替代第二次归一化，从而提升了效率。实证结果表明，在预训练过程中，BHyT 表现出更好的稳定性和效率，与 RMSNorm 相比，训练速度平均提升 15.8%，令牌生成吞吐量平均提高 4.2%，同时在语言理解和推理基准测试中，其推理性能和鲁棒性与之相当或更优。我们的代码公开于：https://anonymous.4open.science/r/BHyT