In this paper, we primarily focus on understanding the data preprocessing pipeline for DNN Training in the public cloud. First, we run experiments to test the performance implications of the two major data preprocessing methods using either raw data or record files. The preliminary results show that data preprocessing is a clear bottleneck, even with the most efficient software and hardware configuration enabled by NVIDIA DALI, a high-optimized data preprocessing library. Second, we identify the potential causes, exercise a variety of optimization methods, and present their pros and cons. We hope this work will shed light on the new co-design of ``data storage, loading pipeline'' and ``training framework'' and flexible resource configurations between them so that the resources can be fully exploited and performance can be maximized.

翻译：本文主要关注在公有云环境下深度神经网络训练的数据预处理流程的理解。首先，我们通过实验测试了使用原始数据或记录文件这两种主要数据预处理方法对性能的影响。初步结果表明，即使采用由高度优化的数据预处理库NVIDIA DALI所支持的最高效软硬件配置，数据预处理仍然是明显的瓶颈。其次，我们识别了潜在原因，尝试了多种优化方法，并阐述了各自的优缺点。我们期望这项工作能为“数据存储、加载管道”与“训练框架”的新协同设计，以及两者之间灵活的资源配置提供启示，从而充分利用资源并最大化性能。