The performance of a lifelong learning (L3) model degrades when it is trained on a series of tasks, as the geometrical formation of the embedding space changes while learning novel concepts sequentially. The majority of existing L3 approaches operate on a fixed-curvature (e.g., zero-curvature Euclidean) space that is not necessarily suitable for modeling the complex geometric structure of data. Furthermore, the distillation strategies apply constraints directly on low-dimensional embeddings, discouraging the L3 model from learning new concepts by making the model highly stable. To address the problem, we propose a distillation strategy named L3DMC that operates on mixed-curvature spaces to preserve the already-learned knowledge by modeling and maintaining complex geometrical structures. We propose to embed the projected low dimensional embedding of fixed-curvature spaces (Euclidean and hyperbolic) to higher-dimensional Reproducing Kernel Hilbert Space (RKHS) using a positive-definite kernel function to attain rich representation. Afterward, we optimize the L3 model by minimizing the discrepancies between the new sample representation and the subspace constructed using the old representation in RKHS. L3DMC is capable of adapting new knowledge better without forgetting old knowledge as it combines the representation power of multiple fixed-curvature spaces and is performed on higher-dimensional RKHS. Thorough experiments on three benchmarks demonstrate the effectiveness of our proposed distillation strategy for medical image classification in L3 settings. Our code implementation is publicly available at https://github.com/csiro-robotics/L3DMC.

翻译：摘要：当终身学习（L3）模型在一系列任务上进行训练时，其性能会下降，这是因为在顺序学习新概念的过程中，嵌入空间的几何结构发生了变化。现有的大多数L3方法在固定曲率（如零曲率的欧几里得）空间上运行，而这种空间不一定适用于建模数据的复杂几何结构。此外，蒸馏策略直接对低维嵌入施加约束，通过使模型高度稳定来阻碍L3模型学习新概念。为解决这一问题，我们提出了一种名为L3DMC的蒸馏策略，该策略在混合曲率空间上操作，通过建模和维持复杂的几何结构来保留已学知识。我们提出使用正定核函数将固定曲率空间（欧几里得和双曲）的投影低维嵌入映射到高维再生核希尔伯特空间（RKHS），以获得丰富的表示。随后，通过最小化新样本表示与旧表示在RKHS中构建的子空间之间的差异来优化L3模型。L3DMC能够更好地适应新知识而不会遗忘旧知识，因为它结合了多个固定曲率空间的表示能力，并在高维RKHS上执行。在三个基准上的全面实验证明了我们提出的蒸馏策略在L3设置下用于医学图像分类的有效性。我们的代码实现公开在https://github.com/csiro-robotics/L3DMC。