Deep Neural Networks provide state-of-the-art accuracy for vision tasks but they require significant resources for training. Thus, they are trained on cloud servers far from the edge devices that acquire the data. This issue increases communication cost, runtime and privacy concerns. In this study, a novel hierarchical training method for deep neural networks is proposed that reduces the communication cost, training runtime, and privacy concerns by dividing the architecture between edge and cloud workers using early exits. The method proposes a brand-new use case for early exits to separate the backward pass of neural networks between the edge and the cloud during the training phase. We address the issues of most available hierarchical training methods that due to the sequential nature of the training phase, cannot train the levels of hierarchy at the same time or they do it with the cost of privacy. In contrast to these schemes, our method can use both edge and cloud workers simultaneously, does not share the raw input data with the cloud, and does not require communication during the backward pass. Several simulations and on-device experiments for different neural network architectures are done to demonstrate the effectiveness of this method. It is shown that the method reduces 29% and 61% runtime in CIFAR-10 classification experiment for VGG-16 and ResNet-18 when the communication with the cloud is done over the 3G protocol. This gain in the runtime is achieved whilst the accuracy drop is negligible. This method can be inspirational to provide online learning of high-accuracy deep neural networks on low-resource devices such as mobile phones or robots as a part of an edge-cloud system, making them more flexible in facing new tasks and classes of data in the future.

翻译：深度神经网络在视觉任务中实现了最先进的精度，但其训练需要大量资源。因此，这些网络通常部署在远离数据采集边缘设备的云服务器上，由此带来通信成本、运行时及隐私问题的增加。本研究提出一种新颖的深度神经网络分层训练方法，通过利用提前退出机制将网络架构划分为边缘端与云端工作节点，从而降低通信成本、训练运行时及隐私风险。该方法开创性地将提前退出机制用于训练阶段，实现神经网络反向传播在边缘端与云端之间的分离。现有分层训练方法因训练阶段固有的顺序性而无法同时训练各层级，或需以隐私为代价实现并行训练。与现有方案不同，本方法可同时利用边缘端与云端工作节点，既不向云端共享原始输入数据，亦无需在反向传播期间进行通信。通过多种神经网络架构的仿真实验及设备端实验验证了该方法的有效性。实验表明，在基于3G协议的云端通信条件下，该方法在CIFAR-10分类任务中使VGG-16和ResNet-18的运行时分别降低29%和61%，同时精度损失可忽略不计。该方法可启发未来在手机、机器人等低资源设备上作为边缘-云端系统组成部分实现高精度深度神经网络的在线学习，使其在应对新型任务与数据类别时更具灵活性。