Deep learning datasets are expanding at an unprecedented pace, creating new challenges for data processing in model training pipelines. A crucial aspect of these pipelines is dataset shuffling, which significantly improves unbiased learning and convergence accuracy by adhering to the principles of random sampling. However, loading shuffled data for large datasets incurs significant overhead in the deep learning pipeline and severely impacts the end-to-end training throughput. To mitigate this, current deep learning systems often resort to partial dataset shuffling, sacrificing global randomness to maintain acceptable training throughput on large datasets, still leaving global shuffling efficiency issues not fully explored. In this work, we present RINAS, a data loading framework that systematically addresses the performance bottleneck of loading global shuffled datasets. Our key contribution is to offer an intra-batch unordered data fetching approach, which unleashes unexplored parallelism of data loading. We implement RINAS under the PyTorch framework for common dataset libraries HuggingFace and TorchVision. Our experimental results show that RINAS improves the throughput of general language model training and vision model training by up to 59% and 89%, respectively.

翻译：深度学习数据集正以前所未有的速度扩张，为模型训练流程中的数据处理带来了新的挑战。数据加载流程的关键环节是数据集随机打乱（shuffling），其遵循随机采样原理，能显著提升无偏学习效果与收敛精度。然而，为大规模数据集加载随机打乱后的数据会产生显著开销，严重制约端到端训练吞吐量。为此，当前深度学习系统常采用部分随机打乱策略，在保证大规模数据集可接受训练吞吐量的同时牺牲全局随机性，使得全局打乱效率问题仍未得到充分探索。本文提出RINAS——一种系统性解决全局随机打乱数据集加载性能瓶颈的数据框架。我们的核心贡献在于提出批内无序数据抓取方法，释放了数据加载中尚未被挖掘的并行潜力。我们在PyTorch框架下针对HuggingFace和TorchVision等通用数据集库实现了RINAS。实验结果表明，RINAS可将通用语言模型训练和视觉模型训练的吞吐量分别提升高达59%和89%。