An ultimate objective in continual learning is to preserve knowledge learned in preceding tasks while learning new tasks. To mitigate forgetting prior knowledge, we propose a novel knowledge distillation technique that takes into the account the manifold structure of the latent/output space of a neural network in learning novel tasks. To achieve this, we propose to approximate the data manifold up-to its first order, hence benefiting from linear subspaces to model the structure and maintain the knowledge of a neural network while learning novel concepts. We demonstrate that the modeling with subspaces provides several intriguing properties, including robustness to noise and therefore effective for mitigating Catastrophic Forgetting in continual learning. We also discuss and show how our proposed method can be adopted to address both classification and segmentation problems. Empirically, we observe that our proposed method outperforms various continual learning methods on several challenging datasets including Pascal VOC, and Tiny-Imagenet. Furthermore, we show how the proposed method can be seamlessly combined with existing learning approaches to improve their performances. The codes of this article will be available at https://github.com/csiro-robotics/SDCL.

翻译：持续学习的一个终极目标是在学习新任务的同时保留先前任务中习得的知识。为了减轻对先前知识的遗忘，我们提出了一种新颖的知识蒸馏技术，该技术在学习新任务时考虑了神经网络潜在/输出空间的流形结构。为实现这一目标，我们建议将数据流形近似到其一阶结构，从而利用线性子空间来建模神经网络的结构并维持其知识，同时学习新概念。我们证明，利用子空间进行建模具有多种引人注目的特性，包括对噪声的鲁棒性，因此能有效缓解持续学习中的“灾难性遗忘”。我们还讨论并展示了如何将所提出的方法用于解决分类和分割问题。实验结果表明，在包括Pascal VOC和Tiny-ImageNet等多个具有挑战性的数据集上，我们的方法优于多种持续学习方法。此外，我们展示了该方法如何与现有的学习方法无缝结合，以提升其性能。本文代码将发布于https://github.com/csiro-robotics/SDCL。