Sequence learning is dominated by Transformers and parallelizable recurrent neural networks (RNNs) such as state-space models, yet learning long-term dependencies remains challenging, and state-of-the-art designs trade power consumption for performance. The Bistable Memory Recurrent Unit (BMRU) was introduced to enable hardware-software co-design of ultra-low power RNNs: quantized states with hysteresis provide persistent memory while mapping directly to analog primitives. However, BMRU performance lags behind parallelizable RNNs on complex sequential tasks. In this paper, we identify gradient blocking during state updates as a key limitation and propose a cumulative update formulation that restores gradient flow while preserving persistent memory, creating skip-connections through time. This leads to the Cumulative Memory Recurrent Unit (CMRU) and its relaxed variant, the $α$CMRU. Experiments show that the cumulative formulation dramatically improves convergence stability and reduces initialization sensitivity. The CMRU and $α$CMRU match or outperform Linear Recurrent Units (LRUs) and minimal Gated Recurrent Units (minGRUs) across diverse benchmarks at small model sizes, with particular advantages on tasks requiring discrete long-range retention, while the CMRU retains quantized states, persistent memory, and noise-resilient dynamics essential for analog implementation.

翻译：序列学习目前主要由Transformer及状态空间模型等并行可循环神经网络主导，然而长期依赖的学习仍具挑战性，且现有设计往往以功耗换取性能。双稳态记忆循环单元通过引入具有滞回特性的量化状态实现持久记忆，并可直接映射至模拟基元，从而支持超低功耗RNN的软硬件协同设计。但BMRU在复杂序列任务上的表现落后于并行化RNN。本文发现状态更新过程中的梯度阻断是关键限制因素，提出一种累积更新公式，在保持持久记忆的同时恢复梯度流动，并创建跨时间步的跳跃连接。该方案衍生出累积记忆循环单元及其松弛变体αCMRU。实验表明，累积公式显著提升了收敛稳定性并降低了对初始化的敏感性。在多种基准测试的小规模模型中，CMRU与αCMRU的性能匹配或超越线性循环单元与最小门控循环单元，特别是在需要离散长距离保持的任务中优势明显，同时CMRU保留了量化状态、持久记忆及对模拟实现至关重要的抗噪动力学特性。