Training recurrent neural networks is known to be difficult when time dependencies become long. In this work, we show that most standard cells only have one stable equilibrium at initialisation, and that learning on tasks with long time dependencies generally occurs once the number of network stable equilibria increases; a property known as multistability. Multistability is often not easily attained by initially monostable networks, making learning of long time dependencies between inputs and outputs difficult. This insight leads to the design of a novel way to initialise any recurrent cell connectivity through a procedure called "warmup" to improve its capability to learn arbitrarily long time dependencies. This initialisation procedure is designed to maximise network reachable multistability, i.e., the number of equilibria within the network that can be reached through relevant input trajectories, in few gradient steps. We show on several information restitution, sequence classification, and reinforcement learning benchmarks that warming up greatly improves learning speed and performance, for multiple recurrent cells, but sometimes impedes precision. We therefore introduce a double-layer architecture initialised with a partial warmup that is shown to greatly improve learning of long time dependencies while maintaining high levels of precision. This approach provides a general framework for improving learning abilities of any recurrent cell when long time dependencies are present. We also show empirically that other initialisation and pretraining procedures from the literature implicitly foster reachable multistability of recurrent cells.

翻译：训练循环神经网络在处理长时间依赖时已知非常困难。本研究表明，大多数标准单元在初始化时仅有一个稳定平衡点，而涉及长时间依赖的任务学习通常发生在网络稳定平衡点数量增加时——这种性质被称为多稳态性。初始单稳态网络往往难以获得多稳态性，导致输入输出之间的长时间依赖学习困难。这一发现引导我们设计了一种新型初始化方法，通过称为"预热"（warmup）的过程初始化任意循环单元连接，以提升其学习任意长时间依赖的能力。该初始化过程旨在通过少量梯度步数最大化网络的可达多稳态性，即网络内可通过相关输入轨迹到达的平衡点数量。我们在多个信息恢复、序列分类和强化学习基准上证明，预热能显著提升多种循环单元的学习速度与性能，但有时会牺牲精度。为此，我们提出一种采用部分预热初始化的双层架构，既能大幅提升长时间依赖的学习能力，又能保持高精度。该方法为存在长时间依赖时提升任意循环单元学习能力提供了通用框架。实验还表明，文献中的其他初始化与预训练方法本质上都隐式促进了循环单元的可达多稳态性。