Sequence learning is dominated by Transformers and parallelizable recurrent neural networks (RNNs) such as state-space models, yet learning long-term dependencies remains challenging, and state-of-the-art designs trade power consumption for performance. The Bistable Memory Recurrent Unit (BMRU) was introduced to enable hardware-software co-design of ultra-low power RNNs: quantized states with hysteresis provide persistent memory while mapping directly to analog primitives. However, BMRU performance lags behind parallelizable RNNs on complex sequential tasks. In this paper, we identify gradient blocking during state updates as a key limitation and propose a cumulative update formulation that restores gradient flow while preserving persistent memory, creating skip-connections through time. This leads to the Cumulative Memory Recurrent Unit (CMRU) and its relaxed variant, the $α$CMRU. Experiments show that the cumulative formulation dramatically improves convergence stability and reduces initialization sensitivity. The CMRU and $α$CMRU match or outperform Linear Recurrent Units (LRUs) and minimal Gated Recurrent Units (minGRUs) across diverse benchmarks at small model sizes, with particular advantages on tasks requiring discrete long-range retention, while the CMRU retains quantized states, persistent memory, and noise-resilient dynamics essential for analog implementation.

翻译：序列学习主要由Transformer和可并行化循环神经网络（RNN，如状态空间模型）主导，然而长程依赖学习仍具挑战性，且现有最先进设计通常以功耗换取性能。双稳态记忆循环单元（BMRU）旨在实现超低功耗RNN的软硬件协同设计：具有迟滞特性的量化状态可直接映射至模拟基元，从而提供持久记忆。但BMRU在复杂序列任务上的性能仍落后于可并行化RNN。本文中，我们识别出状态更新过程中的梯度阻断为核心限制因素，并提出一种累积更新公式，该公式在保持持久记忆的同时恢复梯度流，创建贯穿时间的跳跃连接。由此衍生出累积记忆循环单元（CMRU）及其松弛变体 $α$CMRU。实验表明，累积公式显著提升收敛稳定性并降低初始化敏感度。在多种基准测试中，CMRU与$α$CMRU在小型模型规模下匹配或优于线性循环单元（LRU）和最小门控循环单元（minGRU），尤其在对离散长程保持任务中展现优势。同时，CMRU保持了模拟实现所必需的量化状态、持久记忆及抗噪动力学特性。