With the wide adoption of AI applications, there is a pressing need of enabling real-time neural network (NN) inference on small embedded devices, but deploying NNs and achieving high performance of NN inference on these small devices is challenging due to their extremely weak capabilities. Although NN partitioning and offloading can contribute to such deployment, they are incapable of minimizing the local costs at embedded devices. Instead, we suggest to address this challenge via agile NN offloading, which migrates the required computations in NN offloading from online inference to offline learning. In this paper, we present AgileNN, a new NN offloading technique that achieves real-time NN inference on weak embedded devices by leveraging eXplainable AI techniques, so as to explicitly enforce feature sparsity during the training phase and minimize the online computation and communication costs. Experiment results show that AgileNN's inference latency is >6x lower than the existing schemes, ensuring that sensory data on embedded devices can be timely consumed. It also reduces the local device's resource consumption by >8x, without impairing the inference accuracy.

翻译：随着人工智能应用的广泛普及，在小型嵌入式设备上实现实时神经网络推理的需求日益迫切。然而，由于这些设备计算能力极弱，在其上部署神经网络并实现高性能推理极具挑战性。尽管神经网络分区与卸载技术可辅助此类部署，但现有方法无法最小化嵌入式设备的本地开销。为此，我们提出通过敏捷神经网络卸载应对这一挑战——将神经网络卸载所需的计算从在线推理迁移至离线学习阶段。本文提出AgileNN，一种利用可解释AI技术的新型神经网络卸载方法：通过在训练阶段显式施加特征稀疏性约束，最小化在线推理的计算与通信开销，从而在弱嵌入式设备上实现实时神经网络推理。实验结果表明，AgileNN的推理延迟比现有方案降低超过6倍，确保嵌入式设备传感器数据可被及时处理。同时，该方法在保证推理精度不受损的前提下，将本地设备资源消耗降低超过8倍。