Intrinsic capability to continuously learn a changing data stream is a desideratum of deep neural networks (DNNs). However, current DNNs suffer from catastrophic forgetting, which hinders remembering past knowledge. To mitigate this issue, existing Continual Learning (CL) approaches either retain exemplars for replay, regularize learning, or allocate dedicated capacity for new tasks. This paper investigates an unexplored CL direction for incremental learning called Incremental Latent Rectification or ILR. In a nutshell, ILR learns to propagate with correction (or rectify) the representation from the current trained DNN backward to the representation space of the old task, where performing predictive decisions is easier. This rectification process only employs a chain of small representation mapping networks, called rectifier units. Empirical experiments on several continual learning benchmarks, including CIFAR10, CIFAR100, and Tiny ImageNet, demonstrate the effectiveness and potential of this novel CL direction compared to existing representative CL methods.

翻译：深度神经网络（DNNs）持续学习变化数据流的固有能力是其理想特性。然而，当前的DNNs存在灾难性遗忘问题，这阻碍了对过去知识的记忆。为缓解此问题，现有的持续学习（CL）方法要么保留样本进行回放，要么对学习过程进行正则化，要么为新增任务分配专用容量。本文研究了一种名为增量潜在校正（ILR）的、尚未被探索的增量学习CL方向。简而言之，ILR学习通过校正（或修正）的方式，将当前训练好的DNN的表征反向传播到旧任务的表征空间中，在该空间中进行预测决策更为容易。此校正过程仅使用一系列称为校正器单元的小型表征映射网络。在多个持续学习基准（包括CIFAR10、CIFAR100和Tiny ImageNet）上的实证实验表明，与现有的代表性CL方法相比，这一新颖的CL方向具有有效性和潜力。