Plasticity, the ability of a neural network to quickly change its predictions in response to new information, is essential for the adaptability and robustness of deep reinforcement learning systems. Deep neural networks are known to lose plasticity over the course of training even in relatively simple learning problems, but the mechanisms driving this phenomenon are still poorly understood. This paper conducts a systematic empirical analysis into plasticity loss, with the goal of understanding the phenomenon mechanistically in order to guide the future development of targeted solutions. We find that loss of plasticity is deeply connected to changes in the curvature of the loss landscape, but that it often occurs in the absence of saturated units. Based on this insight, we identify a number of parameterization and optimization design choices which enable networks to better preserve plasticity over the course of training. We validate the utility of these findings on larger-scale RL benchmarks in the Arcade Learning Environment.

翻译：可塑性，即神经网络根据新信息快速调整其预测的能力，对于深度强化学习系统的适应性和鲁棒性至关重要。已知深度神经网络即使在相对简单的学习问题中，也会在训练过程中丧失可塑性，但目前对此现象的驱动机制仍知之甚少。本文对可塑性损失进行了系统的实证分析，旨在从机理上理解这一现象，以指导未来针对性解决方案的开发。我们发现可塑性损失与损失景观曲率的变化密切相关，但其往往发生在无饱和神经元的情况下。基于这一认识，我们确定了一系列参数化与优化设计选择，使网络能够在训练过程中更好地保持可塑性。我们通过在Arcade学习环境中的大规模强化学习基准测试，验证了这些发现的实用性。