Recurrent Neural Networks (RNNs) are renowned for their adeptness in modeling temporal dependencies, a trait that has driven their widespread adoption for sequential data processing. Nevertheless, vanilla RNNs are confronted with the well-known issue of gradient vanishing and exploding, posing a significant challenge for learning and establishing long-range dependencies. Additionally, gated RNNs tend to be over-parameterized, resulting in poor network generalization. To address these challenges, we propose a novel Delayed Memory Unit (DMU) in this paper, wherein a delay line structure, coupled with delay gates, is introduced to facilitate temporal interaction and temporal credit assignment, so as to enhance the temporal modeling capabilities of vanilla RNNs. Particularly, the DMU is designed to directly distribute the input information to the optimal time instant in the future, rather than aggregating and redistributing it over time through intricate network dynamics. Our proposed DMU demonstrates superior temporal modeling capabilities across a broad range of sequential modeling tasks, utilizing considerably fewer parameters than other state-of-the-art gated RNN models in applications such as speech recognition, radar gesture recognition, ECG waveform segmentation, and permuted sequential image classification.

翻译：循环神经网络（RNN）因其擅长建模时间依赖性而闻名，这一特性推动了它们在序列数据处理中的广泛应用。然而，传统的RNN面临梯度消失和梯度爆炸这一众所周知的问题，给学习与建立长距离依赖关系带来了巨大挑战。此外，门控RNN往往参数过多，导致网络泛化能力较差。为解决这些挑战，本文提出了一种新颖的延迟记忆单元（DMU），其中引入延迟线结构与延迟门，以促进时间交互与时间信用分配，从而增强传统RNN的时间建模能力。特别地，DMU旨在将输入信息直接分配到未来的最佳时间点，而非通过复杂的网络动态随时间聚合与重新分配。我们提出的DMU在一系列广泛的序列建模任务中展现出卓越的时间建模能力，在语音识别、雷达手势识别、心电波形分割和排列序列图像分类等应用中，使用的参数远少于其他先进的门控RNN模型。