Well-trained deep neural networks (DNNs) treat all test samples equally during prediction. Adaptive DNN inference with early exiting leverages the observation that some test examples can be easier to predict than others. This paper presents EENet, a novel early-exiting scheduling framework for multi-exit DNN models. Instead of having every sample go through all DNN layers during prediction, EENet learns an early exit scheduler, which can intelligently terminate the inference earlier for certain predictions, which the model has high confidence of early exit. As opposed to previous early-exiting solutions with heuristics-based methods, our EENet framework optimizes an early-exiting policy to maximize model accuracy while satisfying the given per-sample average inference budget. Extensive experiments are conducted on four computer vision datasets (CIFAR-10, CIFAR-100, ImageNet, Cityscapes) and two NLP datasets (SST-2, AgNews). The results demonstrate that the adaptive inference by EENet can outperform the representative existing early exit techniques. We also perform a detailed visualization analysis of the comparison results to interpret the benefits of EENet.

翻译：经过良好训练的深度神经网络（DNN）在预测过程中对所有测试样本采取相同的处理策略。基于早期退出的自适应DNN推理利用了一个观察现象：某些测试样本的预测难度可能低于其他样本。本文提出EENet——一种面向多出口DNN模型的新型早期退出调度框架。不同于让所有样本在推理过程中必须遍历全部网络层，EENet通过学习一个早期退出调度器，能够智能地为模型具有高置信度的某些预测提前终止推理过程。与以往基于启发式方法的早期退出方案不同，我们的EENet框架优化了早期退出策略，在满足给定每个样本平均推理预算的前提下最大化模型精度。我们在四个计算机视觉数据集（CIFAR-10、CIFAR-100、ImageNet、Cityscapes）和两个自然语言处理数据集（SST-2、AgNews）上进行了大量实验。结果表明，EENet的自适应推理方法能够超越现有代表性的早期退出技术。我们还对对比结果进行了详细的可视化分析，以阐释EENet的优势所在。