The loss of plasticity in learning agents, analogous to the solidification of neural pathways in biological brains, significantly impedes learning and adaptation in reinforcement learning due to its non-stationary nature. To address this fundamental challenge, we propose a novel approach, {\it Neuroplastic Expansion} (NE), inspired by cortical expansion in cognitive science. NE maintains learnability and adaptability throughout the entire training process by dynamically growing the network from a smaller initial size to its full dimension. Our method is designed with three key components: (\textit{1}) elastic topology generation based on potential gradients, (\textit{2}) dormant neuron pruning to optimize network expressivity, and (\textit{3}) neuron consolidation via experience review to strike a balance in the plasticity-stability dilemma. Extensive experiments demonstrate that NE effectively mitigates plasticity loss and outperforms state-of-the-art methods across various tasks in MuJoCo and DeepMind Control Suite environments. NE enables more adaptive learning in complex, dynamic environments, which represents a crucial step towards transitioning deep reinforcement learning from static, one-time training paradigms to more flexible, continually adapting models.

翻译：学习智能体中的可塑性丧失，类似于生物大脑中神经通路的固化，由于其非平稳特性，显著阻碍了强化学习中的学习与适应能力。为应对这一根本性挑战，我们受认知科学中皮层扩展的启发，提出了一种新颖方法——神经可塑性扩展（NE）。NE通过将网络从较小的初始规模动态增长至完整维度，在整个训练过程中持续保持可学习性与适应性。我们的方法设计包含三个关键组成部分：（1）基于势梯度的弹性拓扑生成，（2）通过休眠神经元剪枝优化网络表达能力，以及（3）通过经验回放进行神经元整合，以在可塑性-稳定性困境中取得平衡。大量实验表明，NE能有效缓解可塑性丧失，并在MuJoCo和DeepMind Control Suite环境中的多种任务上超越现有先进方法。NE使得在复杂动态环境中实现更具适应性的学习成为可能，这标志着深度强化学习从静态的一次性训练范式向更灵活、持续适应模型转变的关键一步。