Continual Semantic Segmentation (CSS) extends static semantic segmentation by incrementally introducing new classes for training. To alleviate the catastrophic forgetting issue in CSS, a memory buffer that stores a small number of samples from the previous classes is constructed for replay. However, existing methods select the memory samples either randomly or based on a single-factor-driven handcrafted strategy, which has no guarantee to be optimal. In this work, we propose a novel memory sample selection mechanism that selects informative samples for effective replay in a fully automatic way by considering comprehensive factors including sample diversity and class performance. Our mechanism regards the selection operation as a decision-making process and learns an optimal selection policy that directly maximizes the validation performance on a reward set. To facilitate the selection decision, we design a novel state representation and a dual-stage action space. Our extensive experiments on Pascal-VOC 2012 and ADE 20K datasets demonstrate the effectiveness of our approach with state-of-the-art (SOTA) performance achieved, outperforming the second-place one by 12.54% for the 6stage setting on Pascal-VOC 2012.

翻译：持续语义分割（CSS）在静态语义分割基础上通过逐步引入新类别进行训练。为缓解CSS中的灾难性遗忘问题，通常构建一个存储少量先前类别样本的记忆缓冲区用于重放。然而，现有方法要么随机选择记忆样本，要么基于单因素驱动的启发式策略，这无法保证最优性。本文提出一种新型记忆样本选择机制，通过综合考虑样本多样性与类别性能等多维因素，以全自动方式选取信息性样本实现高效重放。该机制将选择操作视为决策过程，学习一个直接最大化奖励集验证性能的最优选择策略。为辅助选择决策，我们设计了新型状态表征与双阶段动作空间。在Pascal-VOC 2012和ADE 20K数据集上的大量实验表明，本方法实现了当前最优性能，在Pascal-VOC 2012六阶段设定下以12.54%的绝对优势超越第二名。