Training massive-scale deep learning models on datasets spanning tens of terabytes presents critical challenges in hardware utilization and training reproducibility. In this paper, we identify and resolve profound data-loading bottlenecks within distributed GPU training pipelines using the Petastorm data loader and Apache Parquet datasets. Through systematic profiling, we demonstrate that network I/O and CPU-bound data transformations (e.g., PyArrow to NumPy) constrain GPU utilization to as low as 10-15%. To address this, we propose an optimized architecture that features push-down worker-level transformations coupled with local-disk caching via Fanout-Cache, minimizing redundant I/O and CPU overhead across training epochs. Furthermore, we eliminate race conditions in multi-worker shared queues by implementing dedicated round-robin ventilator and result queues, alongside modernized RNG handling, achieving strict deterministic data loading. Our optimizations yield a 6x speedup, reducing end-to-end training time from 22 hours to 3 hours, increasing GPU utilization to over 60%, and drastically reducing run-to-run variance, enabling robust, high-throughput, and reproducible large-scale model training.

翻译：在涵盖数十TB数据集的超大规模深度学习模型训练中，硬件利用效率与训练可复现性面临严峻挑战。本文通过Petastorm数据加载器与Apache Parquet数据集，识别并解决了分布式GPU训练流水线中的深层数据加载瓶颈。基于系统性性能分析，我们证实网络I/O与CPU密集型数据转换（如PyArrow到NumPy的转换）将GPU利用率限制在10-15%。为此，我们提出一种优化架构，通过Fanout-Cache实现推送式工作节点级转换与本地磁盘缓存的协同设计，将训练轮次中冗余I/O与CPU开销降至最低。通过实现专用轮询式通风器队列与结果队列，并升级随机数生成器处理机制，我们消除了多工作者共享队列中的竞态条件，实现了严格确定性的数据加载。优化方案带来6倍加速比，将端到端训练时间从22小时压缩至3小时，GPU利用率提升至60%以上，训练间方差显著降低，最终构建了兼具高吞吐量、鲁棒性与可复现性的大规模模型训练体系。