Incremental learning (IL) suffers from catastrophic forgetting of old tasks when learning new tasks. This can be addressed by replaying previous tasks' data stored in a memory, which however is usually prone to size limits and privacy leakage. Recent studies store only class centroids as prototypes and augment them with Gaussian noises to create synthetic data for replay. However, they cannot effectively avoid class interference near their margins that leads to forgetting. Moreover, the injected noises distort the rich structure between real data and prototypes, hence even detrimental to IL. In this paper, we propose YONO that You Only Need to replay One condensed prototype per class, which for the first time can even outperform memory-costly exemplar-replay methods. To this end, we develop a novel prototype learning method that (1) searches for more representative prototypes in high-density regions by an attentional mean-shift algorithm and (2) moves samples in each class to their prototype to form a compact cluster distant from other classes. Thereby, the class margins are maximized, which effectively reduces interference causing future forgetting. In addition, we extend YONO to YONO+, which creates synthetic replay data by random sampling in the neighborhood of each prototype in the representation space. We show that the synthetic data can further improve YONO. Extensive experiments on IL benchmarks demonstrate the advantages of YONO/YONO+ over existing IL methods in terms of both accuracy and forgetting.

翻译：增量学习（IL）在学习新任务时面临旧任务灾难性遗忘的问题。这一问题可通过重放存储在内存中的先前任务数据来解决，但内存通常受限于大小和隐私泄露风险。近期研究将每个类别的类心存储为原型，并通过添加高斯噪声生成合成数据进行重放。然而，这些方法无法有效避免类别边界附近的干扰，从而引发遗忘。此外，注入的噪声会扭曲真实数据与原型之间的丰富结构，甚至对增量学习产生负面影响。本文提出YONO方法——每类仅需重放一个压缩原型，首次在性能上超越高内存开销的样本重放方法。为此，我们开发了一种新型原型学习方法：（1）通过注意力均值漂移算法在高密度区域搜索更具代表性的原型；（2）将每类样本向各自原型移动，形成与其他类别距离较远的紧凑簇。通过最大化类别边界，有效减少未来遗忘的干扰。此外，我们将YONO扩展为YONO+，通过在表示空间中每个原型的邻域内随机采样生成合成重放数据。实验表明，合成数据可进一步提升YONO性能。在增量学习基准上的大量实验证明，YONO/YONO+在准确率和遗忘抑制两方面均优于现有增量学习方法。