State-of-the-art handwritten text recognition (HTR) systems commonly use Transformers, whose growing key-value (KV) cache makes decoding slow and memory-intensive. We introduce DRetHTR, a decoder-only model built on Retentive Networks (RetNet). Compared to an equally sized decoder-only Transformer baseline, DRetHTR delivers 1.6-1.9x faster inference with 38-42% less memory usage, without loss of accuracy. By replacing softmax attention with softmax-free retention and injecting multi-scale sequential priors, DRetHTR avoids a growing KV cache: decoding is linear in output length in both time and memory. To recover the local-to-global inductive bias of attention, we propose layer-wise gamma scaling, which progressively enlarges the effective retention horizon in deeper layers. This encourages early layers to model short-range dependencies and later layers to capture broader context, mitigating the flexibility gap introduced by removing softmax. Consequently, DRetHTR achieves best reported test character error rates of 2.26% (IAM-A, en), 1.81% (RIMES, fr), and 3.46% (Bentham, en), and is competitive on READ-2016 (de) with 4.21%. This demonstrates that decoder-only RetNet enables Transformer-level HTR accuracy with substantially improved decoding speed and memory efficiency.

翻译：当前最先进的手写文本识别系统通常使用Transformer模型，但其不断增长的关键值缓存导致解码速度慢且内存占用高。我们提出了DRetHTR，一种基于保留网络构建的仅解码器模型。与同等规模的仅解码器Transformer基线相比，DRetHTR在保持精度不变的情况下，实现了1.6-1.9倍的推理加速，并减少了38-42%的内存使用。通过用无需softmax的保留机制替代softmax注意力，并注入多尺度序列先验，DRetHTR避免了关键值缓存的增长：解码过程在时间和内存上均与输出长度呈线性关系。为了恢复注意力机制从局部到全局的归纳偏置，我们提出了层间伽马缩放策略，该策略在更深层中逐步扩大有效保留范围。这促使浅层建模短程依赖，深层捕获更广泛的上下文，从而缓解因移除softmax而引入的灵活性差距。因此，DRetHTR在多个数据集上取得了目前报道的最佳测试字符错误率：IAM-A（英文）为2.26%，RIMES（法文）为1.81%，Bentham（英文）为3.46%，并在READ-2016（德文）上以4.21%的错误率表现出竞争力。这表明仅解码器的保留网络能够实现Transformer级别的手写文本识别精度，同时显著提升解码速度和内存效率。