Deep neural networks provide state-of-the-art accuracy for vision tasks but they require significant resources for training. Thus, they are trained on cloud servers far from the edge devices that acquire the data. This issue increases communication cost, runtime and privacy concerns. In this study, a novel hierarchical training method for deep neural networks is proposed that uses early exits in a divided architecture between edge and cloud workers to reduce the communication cost, training runtime and privacy concerns. The method proposes a brand-new use case for early exits to separate the backward pass of neural networks between the edge and the cloud during the training phase. We address the issues of most available methods that due to the sequential nature of the training phase, cannot train the levels of hierarchy simultaneously or they do it with the cost of compromising privacy. In contrast, our method can use both edge and cloud workers simultaneously, does not share the raw input data with the cloud and does not require communication during the backward pass. Several simulations and on-device experiments for different neural network architectures demonstrate the effectiveness of this method. It is shown that the proposed method reduces the training runtime by 29% and 61% in CIFAR-10 classification experiment for VGG-16 and ResNet-18 when the communication with the cloud is done at a low bit rate channel. This gain in the runtime is achieved whilst the accuracy drop is negligible. This method is advantageous for online learning of high-accuracy deep neural networks on low-resource devices such as mobile phones or robots as a part of an edge-cloud system, making them more flexible in facing new tasks and classes of data.

翻译：深度神经网络在视觉任务中展现了最先进的精度，但其训练需要大量计算资源。因此，训练过程通常在远离数据采集端设备的云服务器上进行，这带来了通信成本、运行时间和隐私安全方面的问题。本研究提出了一种新颖的深度神经网络分层训练方法，通过在边缘设备与云端服务器之间划分架构并利用早期退出机制，有效降低了通信成本、训练运行时间并缓解了隐私顾虑。该方法为早期退出机制开辟了全新应用场景——在训练阶段将神经网络的反向传播过程分离到边缘和云端分别执行。我们解决了现有方法因训练阶段顺序执行特性而无法同时训练分层网络层级、或以牺牲隐私为前提实现并行训练的缺陷。相比之下，我们的方法能同时利用边缘和云端计算资源，无需向云端共享原始输入数据，且在反向传播过程中不产生通信需求。针对不同神经网络架构的多次仿真与设备端实验验证了该方法的有效性。在CIFAR-10分类任务中，当通过低比特率信道与云端通信时，该方法使VGG-16和ResNet-18的训练运行时间分别降低29%和61%，且精度损失可忽略不计。该技术特别适用于边缘-云系统中手机或机器人等低资源设备的高精度深度神经网络在线学习，使其在面对新任务和新数据类型时具备更强的适应性。