Sparsity is a well-studied technique for compressing deep neural networks (DNNs) without compromising performance. In deep reinforcement learning (DRL), neural networks with up to 5% of their original weights can still be trained with minimal performance loss compared to their dense counterparts. However, most existing methods rely on unstructured fine-grained sparsity, which limits hardware acceleration opportunities due to irregular computation patterns. Structured coarse-grained sparsity enables hardware acceleration, yet typically degrades performance and increases pruning complexity. In this work, we present, to the best of our knowledge, the first study on N:M structured sparsity in RL, which balances compression, performance, and hardware efficiency. Our framework enforces row-wise N:M sparsity throughout training for all networks in off-policy RL (TD3), maintaining compatibility with accelerators that support N:M sparse matrix operations. Experiments on continuous-control benchmarks show that RNM-TD3, our N:M sparse agent, outperforms its dense counterpart at 50%-75% sparsity (e.g., 2:4 and 1:4), achieving up to a 14% increase in performance at 2:4 sparsity on the Ant environment. RNM-TD3 remains competitive even at 87.5% sparsity (1:8), while enabling potential training speedups.

翻译：稀疏性是一种经过深入研究的、可在不损害性能的前提下压缩深度神经网络的技术。在深度强化学习中，仅保留原始权重5%的神经网络，经过训练后，其性能损失相较于稠密网络仍可维持在极低水平。然而，现有方法大多依赖于非结构化的细粒度稀疏性，其不规则的计算模式限制了硬件加速的潜力。结构化的粗粒度稀疏性虽能实现硬件加速，但通常会降低性能并增加剪枝复杂度。在本研究中，据我们所知，我们首次在强化学习领域探讨了N:M结构化稀疏性，它在压缩率、性能和硬件效率之间取得了平衡。我们的框架在离线策略强化学习的所有网络中，全程强制执行行级N:M稀疏性，确保了与支持N:M稀疏矩阵运算的加速器的兼容性。在连续控制基准测试上的实验表明，我们提出的N:M稀疏智能体RNM-TD3，在50%-75%的稀疏度下（例如2:4和1:4），性能优于其稠密版本，例如在Ant环境中，2:4稀疏度下性能提升高达14%。即使在87.5%的稀疏度下（1:8），RNM-TD3仍保持竞争力，同时具备实现训练加速的潜力。