Continual learning (CL) has remained a persistent challenge for deep neural networks due to catastrophic forgetting (CF) of previously learned tasks. Several techniques such as weight regularization, experience rehearsal, and parameter isolation have been proposed to alleviate CF. Despite their relative success, these research directions have predominantly remained orthogonal and suffer from several shortcomings, while missing out on the advantages of competing strategies. On the contrary, the brain continually learns, accommodates, and transfers knowledge across tasks by simultaneously leveraging several neurophysiological processes, including neurogenesis, active forgetting, neuromodulation, metaplasticity, experience rehearsal, and context-dependent gating, rarely resulting in CF. Inspired by how the brain exploits multiple mechanisms concurrently, we propose TriRE, a novel CL paradigm that encompasses retaining the most prominent neurons for each task, revising and solidifying the extracted knowledge of current and past tasks, and actively promoting less active neurons for subsequent tasks through rewinding and relearning. Across CL settings, TriRE significantly reduces task interference and surpasses different CL approaches considered in isolation.

翻译：持续学习（CL）因灾难性遗忘（CF）先前学习任务而始终是深度神经网络的持久挑战。研究者已提出权重正则化、经验重演、参数隔离等多种技术以缓解CF。尽管这些方法取得相对成功，但它们本质上仍保持正交性，存在若干缺陷，并错失了竞争策略的优势。相反，大脑通过同步利用多种神经生理过程——包括神经发生、主动遗忘、神经调节、元可塑性、经验重演和情境依赖性门控，持续学习、适应和迁移跨任务知识，极少出现CF。受大脑并发运用多重机制的启发，我们提出TriRE这一新型CL范式，该范式包含：为每个任务保留最突出神经元、修正并巩固当前与过去任务提取的知识，以及通过回退与重学习主动促进非活跃神经元适应后续任务。在多种CL设置中，TriRE显著减少任务干扰，并超越各类独立考虑的CL方法。