Incremental semantic segmentation aims to continually learn the segmentation of new coming classes without accessing the training data of previously learned classes. However, most current methods fail to address catastrophic forgetting and background shift since they 1) treat all previous classes equally without considering different forgetting paces caused by imbalanced gradient back-propagation; 2) lack strong semantic guidance between classes. To tackle the above challenges, in this paper, we propose a Gradient-Semantic Compensation (GSC) model, which surmounts incremental semantic segmentation from both gradient and semantic perspectives. Specifically, to address catastrophic forgetting from the gradient aspect, we develop a step-aware gradient compensation that can balance forgetting paces of previously seen classes via re-weighting gradient backpropagation. Meanwhile, we propose a soft-sharp semantic relation distillation to distill consistent inter-class semantic relations via soft labels for alleviating catastrophic forgetting from the semantic aspect. In addition, we develop a prototypical pseudo re-labeling that provides strong semantic guidance to mitigate background shift. It produces high-quality pseudo labels for old classes in the background by measuring distances between pixels and class-wise prototypes. Extensive experiments on three public datasets, i.e., Pascal VOC 2012, ADE20K, and Cityscapes, demonstrate the effectiveness of our proposed GSC model.

翻译：增量语义分割旨在持续学习新出现类别的分割任务，而无需访问已学习类别的训练数据。然而，当前大多数方法难以解决灾难性遗忘和背景偏移问题，原因在于：1）对所有先前类别进行同等处理，未考虑因梯度反向传播不平衡导致的不同遗忘速度；2）类别间缺乏强语义引导。针对上述挑战，本文提出梯度-语义补偿（GSC）模型，从梯度与语义两个维度突破增量语义分割的局限性。具体而言，在梯度层面解决灾难性遗忘时，我们开发了步态感知梯度补偿方法，通过重新加权梯度反向传播来平衡先前已见类别的遗忘速度。同时，提出软-锐语义关系蒸馏方法，利用软标签维持类别间一致的语义关系，从语义层面缓解灾难性遗忘。此外，设计原型伪重标注机制，通过测量像素与类别原型之间的距离，为背景中的旧类别生成高质量伪标签，从而提供强语义引导以缓解背景偏移。在Pascal VOC 2012、ADE20K和Cityscapes三个公开数据集上的大量实验表明，本文提出的GSC模型具有显著有效性。