The development of large-scale foundation models, particularly Large Language Models (LLMs), is constrained by significant computational and memory bottlenecks. These challenges elevate throughput optimization from a mere engineering task to a critical strategic lever, directly influencing training time, operational cost, and the feasible scale of next-generation models. This paper synthesizes evidence from recent academic and industry innovations to analyze key advancements in training efficiency. We examine architectural solutions to dataloader bottlenecks, such as the OVERLORD framework, which has demonstrated a 4.5% improvement in end-to-end training throughput. We investigate memory optimization techniques designed to overcome the GPU memory wall, including CPU offloading strategies like DeepSpeed's ZeRO-Offload, which enable the training of models far exceeding single-accelerator capacity. Furthermore, we explore the growing importance of compiler-centric optimizations, exemplified by Triton-distributed, which enables the joint optimization of computation, memory, and communication for substantial performance gains. The analysis is contextualized by advanced profiling tools and hardware characterization studies that identify and mitigate previously overlooked overheads like Dynamic Voltage and Frequency Scaling (DVFS). Findings indicate that a holistic, system-level approach, integrating innovations across data pipelines, memory management, network fabrics, and compiler technologies, is essential for accelerating AI development, managing costs, and pushing the boundaries of model scale.

翻译：大规模基础模型（特别是大型语言模型）的开发受限于显著的计算和内存瓶颈。这些挑战将吞吐量优化从单纯的工程任务提升为关键的战略杠杆,直接影响到训练时间、运营成本以及下一代模型的可扩展规模。本文综合近期学术界和工业界的创新成果,系统分析训练效率的关键进展。我们考察了数据加载器瓶颈的架构解决方案,例如OVERLORD框架已实现端到端训练吞吐量4.5%的提升。我们研究了为突破GPU内存墙而设计的内存优化技术,包括DeepSpeed的ZeRO-Offload等CPU卸载策略,这些技术使得训练远超单加速器容量的模型成为可能。此外,我们探讨了以编译器为中心的优化策略日益增长的重要性,其中Triton-distributed能够实现计算、内存和通信的联合优化以获得显著的性能增益。本分析基于先进的剖析工具和硬件特性研究,这些工具和方法识别并缓解了此前被忽视的开销（如动态电压频率调整）。研究结果表明,采用贯穿数据流水线、内存管理、网络架构和编译技术的全局性系统级创新方法,对于加速AI开发、控制成本以及突破模型规模边界至关重要。