Deep neural networks provide state-of-the-art accuracy for vision tasks but they require significant resources for training. Thus, they are trained on cloud servers far from the edge devices that acquire the data. This issue increases communication cost, runtime and privacy concerns. In this study, a novel hierarchical training method for deep neural networks is proposed that uses early exits in a divided architecture between edge and cloud workers to reduce the communication cost, training runtime and privacy concerns. The method proposes a brand-new use case for early exits to separate the backward pass of neural networks between the edge and the cloud during the training phase. We address the issues of most available methods that due to the sequential nature of the training phase, cannot train the levels of hierarchy simultaneously or they do it with the cost of compromising privacy. In contrast, our method can use both edge and cloud workers simultaneously, does not share the raw input data with the cloud and does not require communication during the backward pass. Several simulations and on-device experiments for different neural network architectures demonstrate the effectiveness of this method. It is shown that the proposed method reduces the training runtime by 29% and 61% in CIFAR-10 classification experiment for VGG-16 and ResNet-18 when the communication with the cloud is done at a low bit rate channel. This gain in the runtime is achieved whilst the accuracy drop is negligible. This method is advantageous for online learning of high-accuracy deep neural networks on low-resource devices such as mobile phones or robots as a part of an edge-cloud system, making them more flexible in facing new tasks and classes of data.

翻译：深度神经网络在视觉任务中实现了最先进的精度，但训练过程需要大量资源，因此通常在与获取数据的边缘设备相距甚远的云服务器上进行训练。这导致了通信成本、运行时延和隐私问题的增加。本研究提出了一种新颖的深度神经网络分层训练方法，通过在边缘与云端工作者之间的分布式架构中利用早期退出机制，以降低通信成本、训练运行时延和隐私风险。该方法为早期退出开辟了全新的应用场景：在训练阶段将神经网络的逆向传播过程分割至边缘端和云端。我们解决了现有方法中因训练阶段的顺序性而无法同时训练分层结构，或是以牺牲隐私为代价实现并行训练的缺陷。相比之下，我们的方法能够同时利用边缘和云端工作者，无需与云端共享原始输入数据，且逆向传播过程中无需通信。针对不同神经网络架构的仿真实验和实际设备测试均证明了该方法的有效性。结果表明，在CIFAR-10分类实验中，当通过低比特率信道与云端通信时，该方法使VGG-16和ResNet-18的训练运行时间分别减少29%和61%，同时准确率下降可忽略不计。该方法适用于移动设备或机器人等资源受限设备上高精度深度神经网络的在线学习场景（作为边缘-云系统的一部分），从而增强其应对新任务和数据类型的灵活性。