Driven by the demand for energy-efficient employment of deep neural networks, early-exit methods have experienced a notable increase in research attention. These strategies allow for swift predictions by making decisions early in the network, thereby conserving computation time and resources. However, so far the early-exit networks have only been developed for stationary data distributions, which restricts their application in real-world scenarios with continuous non-stationary data. This study aims to explore the continual learning of the early-exit networks. We adapt existing continual learning methods to fit with early-exit architectures and investigate their behavior in the continual setting. We notice that early network layers exhibit reduced forgetting and can outperform standard networks even when using significantly fewer resources. Furthermore, we analyze the impact of task-recency bias on early-exit inference and propose Task-wise Logits Correction (TLC), a simple method that equalizes this bias and improves the network performance for every given compute budget in the class-incremental setting. We assess the accuracy and computational cost of various continual learning techniques enhanced with early-exits and TLC across standard class-incremental learning benchmarks such as 10 split CIFAR100 and ImageNetSubset and show that TLC can achieve the accuracy of the standard methods using less than 70\% of their computations. Moreover, at full computational budget, our method outperforms the accuracy of the standard counterparts by up to 15 percentage points. Our research underscores the inherent synergy between early-exit networks and continual learning, emphasizing their practical utility in resource-constrained environments.

翻译：受深度神经网络能效部署需求的推动，早期退出方法的研究关注度显著提升。这类策略通过在网络浅层做出快速预测，节省了计算时间与资源。然而，当前早期退出网络仅针对平稳数据分布设计，限制了其在真实场景持续非平稳数据流中的应用。本研究旨在探索早期退出网络的持续学习能力。我们改编现有持续学习方法以适应早期退出架构，并分析其在持续学习场景中的行为特征。研究发现，网络浅层展现出更低的遗忘率，即便使用显著更少的资源，其性能也可超越标准网络。此外，我们分析了任务近因偏差对早期退出推理的影响，并提出任务级逻辑校正方法（TLC），通过均衡这种偏差，在类增量设定下以任意计算预算均能提升网络性能。我们在标准类增量学习基准（如10任务划分CIFAR100和ImageNetSubset）上，评估了整合早期退出与TLC技术的多种持续学习方法在准确率与计算成本方面的表现。结果表明，TLC方法仅需标准方法不到70%的计算量即可达到相同准确率。在满计算预算下，我们的方法相较标准方法准确率提升高达15个百分点。本研究揭示了早期退出网络与持续学习间的内在协同效应，强调了其在资源受限环境中的实用价值。