Despite rapid progress in large language models (LLMs), the statistical structure of their weights, activations, and gradients-and its implications for initialization, training dynamics, and efficiency-remains largely unexplored. We empirically show that these quantities in LLMs are well modeled by generalized Gaussian (GG) distributions, and introduce a unified, end-to-end optimization framework grounded in this observation. Our contributions are threefold: (1) a GG-based initialization that aligns with trained model statistics, accelerating convergence and improving accuracy; (2) ACT, a progressive activation-constrained training method that reduces redundancy and propagation overhead; and (3) GCT, a gradient-constrained training algorithm that substantially lowers communication cost in distributed training. Experiments across diverse architectures demonstrate consistently smaller, faster models with minimal communication overhead that match or surpass standard baselines. By anchoring LLM optimization in principled statistical modeling, this work advances efficient, scalable, and hardware-aware AI systems.

翻译：尽管大语言模型（LLMs）发展迅速，但其权重、激活值和梯度的统计结构及其对初始化、训练动态和效率的影响仍未得到充分探索。我们通过实证研究表明，LLMs中的这些量值可由广义高斯（GG）分布精确建模，并基于此观察提出了一个统一的端到端优化框架。本研究的贡献包括三个方面：（1）提出一种基于GG的初始化方法，其与训练后模型的统计特性相匹配，能加速收敛并提升精度；（2）ACT方法，一种渐进式激活约束训练技术，可降低冗余度和传播开销；（3）GCT算法，一种梯度约束训练方法，能显著降低分布式训练中的通信成本。跨多种架构的实验表明，所提方法能持续生成更小、更快的模型，在通信开销最小化的同时达到或超越标准基线性能。通过将LLM优化建立在严谨的统计建模基础上，本研究为构建高效、可扩展且硬件感知的人工智能系统提供了新思路。