Learning continually from a stream of non-i.i.d. data is an open challenge in deep learning, even more so when working in resource-constrained environments such as embedded devices. Visual models that are continually updated through supervised learning are often prone to overfitting, catastrophic forgetting, and biased representations. On the other hand, large language models contain knowledge about multiple concepts and their relations, which can foster a more robust, informed and coherent learning process. This work proposes Continual Visual Mapping (CVM), an approach that continually ground vision representations to a knowledge space extracted from a fixed Language model. Specifically, CVM continually trains a small and efficient visual model to map its representations into a conceptual space established by a fixed Large Language Model. Due to their smaller nature, CVM can be used when directly adapting large visual pre-trained models is unfeasible due to computational or data constraints. CVM overcome state-of-the-art continual learning methods on five benchmarks and offers a promising avenue for addressing generalization capabilities in continual learning, even in computationally constrained devices.

翻译：从非独立同分布数据流中持续学习是深度学习领域的一个开放性挑战，在嵌入式设备等资源受限环境中进行此类学习则更具难度。通过监督学习持续更新的视觉模型常常容易出现过拟合、灾难性遗忘和表示偏差等问题。另一方面，大型语言模型包含多种概念及其关系的知识，这有助于实现更鲁棒、更明智且更连贯的学习过程。本研究提出持续视觉映射（CVM）方法，该方法将持续地把视觉表征锚定到从固定语言模型中提取的知识空间。具体而言，CVM持续训练一个轻量高效的视觉模型，将其表征映射到由固定大型语言模型建立的概念空间中。由于其小型化特性，当因计算或数据限制而无法直接适配大型视觉预训练模型时，CVM仍可适用。CVM在五个基准测试中超越了当前最先进的持续学习方法，为解决持续学习中的泛化能力问题提供了有前景的途径，即使在计算受限的设备上也能实现。