Deep learning systems are prone to catastrophic forgetting when learning from a sequence of tasks, where old data from experienced tasks is unavailable when learning from a new task. To mitigate the problem, a line of methods propose to replay the data of experienced tasks when learning new tasks. These methods usually adopt an extra memory to store the data for replay. However, it is not expected in practice considering the memory constraint or data privacy issue. As a replacement, data-free data replay methods are proposed by inverting samples from the classification model. Though achieving good results, these methods still suffer from the inconsistency of the inverted and real training data, which is neglected in the inversion stage in recent works. To that effect, we propose to measure the data consistency quantitatively by some simplification and assumptions. Using the measurement, we analyze existing techniques for inverting samples and get some insightful information that inspires a novel loss function to reduce the inconsistency. Specifically, the loss minimizes the KL divergence of the distributions of inverted and real data under the tied multivariate Gaussian assumption, which is easy to implement in continual learning. In addition, we observe that the norms of old class weights turn to decrease continually as learning progresses. We thus analyze the underlying reasons and propose a simple regularization term to balance the class weights so that the samples of old classes are more distinguishable. To conclude, we propose the Consistency enhanced data replay with debiased classifier for Class Incremental Learning (CCIL). Extensive experiments on CIFAR-100, Tiny-ImageNet, and ImageNet100 show consistently improved performance of CCIL compared to previous approaches.

翻译：深度学习系统在连续任务学习时容易发生灾难性遗忘，因为新任务学习中无法获取先前任务的历史数据。为缓解该问题，一类方法提出在学习新任务时重放历史任务数据。这些方法通常需要额外存储空间保存重放数据，但实际应用中受限于存储约束或数据隐私问题而难以实现。作为替代方案，无数据重放方法通过从分类模型反演样本实现数据重放。尽管取得了良好效果，这类方法仍存在反演数据与真实训练数据不一致的问题，而近期研究在反演阶段往往忽视了该问题。为此，我们通过简化假设对数据一致性进行量化度量。利用该度量方法分析现有样本反演技术，获得启发式见解并提出新型损失函数以降低数据不一致性。具体而言，该损失函数在绑定多元高斯假设下最小化反演数据与真实数据分布的KL散度，易于在持续学习框架中实现。此外，我们观察到旧类别权重的范数会随学习进程持续衰减，通过分析其根本原因，提出简单正则化项均衡各类别权重，使旧类别样本更具区分性。最终提出面向类别增量学习的一致性增强数据重放与去偏分类器方法（CCIL）。在CIFAR-100、Tiny-ImageNet和ImageNet100数据集上的大量实验表明，CCIL相较于已有方法性能获得持续提升。