Efficiently scaling deep neural networks across GPU clusters requires navigating complex trade-offs between computational throughput, memory utilization, and synchronization overhead. This paper presents a unified empirical evaluation of three dominant distributed training paradigms: Distributed Data Parallel (DDP), Fully Sharded Data Parallel (FSDP), and the Parameter Server (PS) architecture. We conduct side-by-side benchmarking on both high-performance (NVIDIA A100) and commodity-class (NVIDIA A10G) clusters to isolate the impact of communication bandwidth and gang-scheduling dependencies. Our results indicate that while DDP achieves a 2-3x speedup in training throughput for standard architectures, FSDP demonstrates a 4-6x reduction in peak memory usage, validating its utility for memory-constrained environments despite higher communication latency. Furthermore, we evaluate the elasticity of the Parameter Server architecture; while Asynchronous PS reduced training time by up to 28% compared to synchronous approaches, it incurred significant accuracy penalties (ranging from 4% to 17%) due to gradient staleness. We also analyze a modified, staleness-mitigating asynchronous protocol, which we found introduced synchronization overheads that negated throughput gains. These findings provide a decision framework for system designers, highlighting that while DDP remains optimal for homogeneous, gang-scheduled clusters, FSDP and PS offer critical alternatives for memory-bound and heterogeneous environments respectively.

翻译：在GPU集群上高效扩展深度神经网络需要在计算吞吐量、内存利用率和同步开销之间进行复杂的权衡。本文对三种主流的分布式训练范式进行了统一的实证评估：分布式数据并行（DDP）、全分片数据并行（FSDP）以及参数服务器（PS）架构。我们在高性能（NVIDIA A100）和商用级（NVIDIA A10G）集群上进行了并行基准测试，以分离通信带宽和组调度依赖性的影响。我们的结果表明，虽然DDP在标准架构上实现了2-3倍的训练吞吐量加速，但FSDP在峰值内存使用上表现出4-6倍的降低，这验证了其在内存受限环境中的实用性，尽管其通信延迟较高。此外，我们评估了参数服务器架构的弹性；虽然异步PS相比同步方法将训练时间减少了高达28%，但由于梯度陈旧性，它导致了显著的精度损失（范围在4%至17%之间）。我们还分析了一种改进的、缓解陈旧性的异步协议，发现其引入的同步开销抵消了吞吐量增益。这些发现为系统设计者提供了一个决策框架，强调尽管DDP在同构、组调度的集群中仍然是最优选择，但FSDP和PS分别为内存受限和异构环境提供了关键替代方案。