Training deep learning models on single-cell datasets with hundreds of millions of cells requires loading data from disk, as these datasets exceed available memory. While random sampling provides the data diversity needed for effective training, it is prohibitively slow due to the random access pattern overhead, whereas sequential streaming achieves high throughput but introduces biases that degrade model performance. We present scDataset, a PyTorch data loader that enables efficient training from on-disk data with seamless integration across diverse storage formats. Our approach combines block sampling and batched fetching to achieve quasi-random sampling that balances I/O efficiency with minibatch diversity. On Tahoe-100M, a dataset of 100 million cells, scDataset achieves more than two orders of magnitude speedup compared to true random sampling while working directly with AnnData files. We provide theoretical bounds on minibatch diversity and empirically show that scDataset matches the performance of true random sampling across multiple classification tasks.

翻译：在包含数亿细胞的大规模单细胞数据集上训练深度学习模型时，由于数据规模超出可用内存容量，必须从磁盘动态加载数据。随机采样虽能提供有效训练所需的数据多样性，但其随机访问模式的开销导致加载速度极其缓慢；而顺序流式读取虽能实现高吞吐量，却会引入降低模型性能的偏差。本文提出scDataset——一个PyTorch数据加载器，能够高效地从磁盘数据进行训练，并实现跨多种存储格式的无缝集成。该方法结合块采样与批量获取技术，实现了在I/O效率与小批量数据多样性之间取得平衡的准随机采样。在包含1亿细胞的Tahoe-100M数据集上，scDataset直接处理AnnData文件时，相比真实随机采样实现了超过两个数量级的加速。我们提供了小批量数据多样性的理论边界，并通过实证研究表明，在多项分类任务中scDataset能够达到与真实随机采样相当的性能表现。