Recurrent Neural Networks (RNNs) have shown great success in modeling time-dependent patterns, but there is limited research on their learned representations of latent temporal features and the emergence of these representations during training. To address this gap, we use timed automata (TA) to introduce a family of supervised learning tasks modeling behavior dependent on hidden temporal variables whose complexity is directly controllable. Building upon past studies from the perspective of dynamical systems, we train RNNs to emulate temporal flipflops, a new collection of TA that emphasizes the need for time-awareness over long-term memory. We find that these RNNs learn in phases: they quickly perfect any time-independent behavior, but they initially struggle to discover the hidden time-dependent features. In the case of periodic "time-of-day" aware automata, we show that the RNNs learn to switch between periodic orbits that encode time modulo the period of the transition rules. We subsequently apply fixed point stability analysis to monitor changes in the RNN dynamics during training, and we observe that the learning phases are separated by a bifurcation from which the periodic behavior emerges. In this way, we demonstrate how dynamical systems theory can provide insights into not only the learned representations of these models, but also the dynamics of the learning process itself. We argue that this style of analysis may provide insights into the training pathologies of recurrent architectures in contexts outside of time-awareness.

翻译：循环神经网络（RNN）在建模时间依赖模式方面取得了显著成功，但关于其对潜在时间特征的学习表征以及这些表征在训练过程中的涌现机制，相关研究仍较为有限。为填补这一空白，我们利用时间自动机（TA）引入了一系列监督学习任务，这些任务可建模依赖于隐藏时间变量的行为，且其复杂度可直接调控。基于以往从动力系统视角开展的研究，我们训练RNN模拟时间触发器——这是一类新型TA集合，其特点在于强调时间感知能力而非长期记忆需求。研究发现，这些RNN呈阶段性学习特征：它们能迅速掌握任何与时间无关的行为，但在初期难以发现隐藏的时间依赖特征。针对周期性"昼夜节律"感知自动机，我们证明RNN能学会在编码时间模转移规则周期的周期性轨道之间切换。随后，我们应用不动点稳定性分析来监测训练过程中RNN动力系统的变化，并观察到学习阶段之间存在分岔现象，周期性行为由此涌现。通过这种方式，我们展示了动力系统理论如何不仅能揭示这些模型的学习表征，还能阐明学习过程本身的动态机制。我们主张，这种分析范式或可为理解循环架构在时间感知领域之外的训练病理学问题提供新见解。