High resource requirement for Deep Neural Network (DNN) training across multiple GPUs necessitates development of various parallelism techniques. In this paper, we introduce two interconnected DNN training frameworks, namely, V-TiMePReSt and I-TiMePReSt, based on pipeline parallelism, a variant of model parallelism. V-TiMePReSt is a completely staleness-free system which enables the DNNs to be trained on the latest updated weights in each stage of all forward and backward passes. Developing staleness-aware systems at the expense of weight stashing reduces GPU-memory consumption, however, increases the number of epochs to converge. Thus, we introduce I-TiMePReSt, which is also a staleness-aware system, but not at the expense of weight stashing. It does not rely solely on the stale weights or the latest updated weights. I-TiMePReSt computes an intermediate weight towards the latter and performs backward pass on it. Additionally, we formulate the significance of the stale weights mathematically depending on the degree of staleness. In contrast to V-TiMePReSt, I-TiMePReSt works based on the assumption that stale weights have a significant contribution in training, which can be quantified mathematically based on the degree of staleness, although there are other contributory factors which should not be ignored. Experimental results show that V-TiMePReSt is advantageous over existing models in terms of $1)$ the extent of staleness of the weight parameter values and $2)$ GPU memory efficiency, while I-TiMePReSt is superior in terms of $1)$ removing staleness of the weight parameters without removing weight stashing and $2)$ maintaining the trade-off between GPU memory consumption and convergence speed (number of epochs).

翻译：在多GPU上进行深度神经网络（DNN）训练的高资源需求催生了多种并行技术的发展。本文基于模型并行的一种变体——流水线并行，介绍了两个相互关联的DNN训练框架，即V-TiMePReSt和I-TiMePReSt。V-TiMePReSt是一个完全无陈旧性的系统，它使得DNN在所有前向和反向传播的每个阶段都能基于最新更新的权重进行训练。以权重暂存为代价开发陈旧性感知系统可以减少GPU内存消耗，但会增加收敛所需的训练轮数。因此，我们引入了I-TiMePReSt，它也是一个陈旧性感知系统，但并非以权重暂存为代价。它不完全依赖于陈旧权重或最新更新的权重。I-TiMePReSt计算一个趋近于后者的中间权重，并基于该权重执行反向传播。此外，我们根据陈旧程度，从数学上公式化了陈旧权重的重要性。与V-TiMePReSt相比，I-TiMePReSt的工作基于以下假设：陈旧权重对训练有重要贡献，这种贡献可以根据陈旧程度进行数学量化，尽管其他影响因素也不应被忽视。实验结果表明，V-TiMePReSt在以下方面优于现有模型：1）权重参数值的陈旧程度；2）GPU内存效率。而I-TiMePReSt则在以下方面表现更优：1）无需移除权重暂存即可消除权重参数的陈旧性；2）在GPU内存消耗与收敛速度（训练轮数）之间保持了良好的权衡。