TeleChat3-MoE is the latest series of TeleChat large language models, featuring a Mixture-of-Experts (MoE) architecture with parameter counts ranging from 105 billion to over one trillion,trained end-to-end on Ascend NPU cluster. This technical report mainly presents the underlying training infrastructure that enables reliable and efficient scaling to frontier model sizes. We detail systematic methodologies for operator-level and end-to-end numerical accuracy verification, ensuring consistency across hardware platforms and distributed parallelism strategies. Furthermore, we introduce a suite of performance optimizations, including interleaved pipeline scheduling, attention-aware data scheduling for long-sequence training,hierarchical and overlapped communication for expert parallelism, and DVM-based operator fusion. A systematic parallelization framework, leveraging analytical estimation and integer linear programming, is also proposed to optimize multi-dimensional parallelism configurations. Additionally, we present methodological approaches to cluster-level optimizations, addressing host- and device-bound bottlenecks during large-scale training tasks. These infrastructure advancements yield significant throughput improvements and near-linear scaling on clusters comprising thousands of devices, providing a robust foundation for large-scale language model development on hardware ecosystems.

翻译：TeleChat3-MoE 是 TeleChat 大语言模型的最新系列，采用混合专家（Mixture-of-Experts, MoE）架构，参数量从 1050 亿到超过一万亿，并在昇腾 NPU 集群上进行了端到端训练。本技术报告主要介绍了支持模型规模可靠且高效扩展至前沿水平的基础训练设施。我们详细阐述了算子级和端到端数值精度验证的系统性方法，以确保跨硬件平台和分布式并行策略的一致性。此外，我们介绍了一系列性能优化技术，包括交错流水线调度、面向长序列训练的注意力感知数据调度、专家并行的分层与重叠通信，以及基于 DVM 的算子融合。我们还提出了一种利用分析估计和整数线性规划的系统性并行化框架，以优化多维度并行配置。另外，我们介绍了集群级优化的方法论，以解决大规模训练任务中主机侧和设备侧的瓶颈。这些基础设施的进步在包含数千个设备的集群上实现了显著的吞吐量提升和接近线性的扩展，为在特定硬件生态上进行大规模语言模型开发奠定了坚实基础。