One of the main motivations of studying continual learning is that the problem setting allows a model to accrue knowledge from past tasks to learn new tasks more efficiently. However, recent studies suggest that the key metric that continual learning algorithms optimize, reduction in catastrophic forgetting, does not correlate well with the forward transfer of knowledge. We believe that the conclusion previous works reached is due to the way they measure forward transfer. We argue that the measure of forward transfer to a task should not be affected by the restrictions placed on the continual learner in order to preserve knowledge of previous tasks. Instead, forward transfer should be measured by how easy it is to learn a new task given a set of representations produced by continual learning on previous tasks. Under this notion of forward transfer, we evaluate different continual learning algorithms on a variety of image classification benchmarks. Our results indicate that less forgetful representations lead to a better forward transfer suggesting a strong correlation between retaining past information and learning efficiency on new tasks. Further, we found less forgetful representations to be more diverse and discriminative compared to their forgetful counterparts.

翻译：持续学习研究的主要动机之一是，该问题设置允许模型从先前任务中积累知识，从而更高效地学习新任务。然而，近期研究表明，持续学习算法优化的关键指标——灾难性遗忘的减少——与知识的前向迁移相关性较弱。我们认为以往研究得出该结论的原因在于其衡量前向迁移的方式。我们主张，对某个任务的前向迁移度量不应受到为保留先前任务知识而对持续学习器施加的限制所影响。相反，前向迁移应通过给定持续学习在先前任务上产生的表征集后，学习新任务的难易程度来衡量。基于这种前向迁移概念，我们在多种图像分类基准上评估了不同的持续学习算法。结果表明，遗忘较少的表征能带来更好的前向迁移，这揭示了保留过去信息与学习新任务效率之间的强相关性。此外，我们发现与易于遗忘的表征相比，遗忘较少的表征更具多样性和判别性。