Reinforcement Learning (RL) provides a powerful framework for decision-making in complex environments. However, implementing RL in hardware-efficient and bio-inspired ways remains a challenge. This paper presents a novel Spiking Neural Network (SNN) architecture for solving RL problems with real-valued observations. The proposed model incorporates multi-layered event-based clustering, with the addition of Temporal Difference (TD)-error modulation and eligibility traces, building upon prior work. An ablation study confirms the significant impact of these components on the proposed model's performance. A tabular actor-critic algorithm with eligibility traces and a state-of-the-art Proximal Policy Optimization (PPO) algorithm are used as benchmarks. Our network consistently outperforms the tabular approach and successfully discovers stable control policies on classic RL environments: mountain car, cart-pole, and acrobot. The proposed model offers an appealing trade-off in terms of computational and hardware implementation requirements. The model does not require an external memory buffer nor a global error gradient computation, and synaptic updates occur online, driven by local learning rules and a broadcasted TD-error signal. Thus, this work contributes to the development of more hardware-efficient RL solutions.

翻译：强化学习为复杂环境中的决策提供了强大框架。然而，以硬件高效且受生物启发的方式实现强化学习仍是一项挑战。本文提出了一种新型脉冲神经网络架构，用于解决实值观测下的强化学习问题。该模型在先前工作基础上，融入多层基于事件的聚类，并引入时序差分误差调制和资格迹。消融研究证实了这些组件对模型性能的显著影响。我们以带有资格迹的表格型演员-评论员算法和先进近端策略优化算法作为基准。我们的网络在经典强化学习环境（山地车、倒立摆和Acrobot）中持续优于表格型方法，并成功发现稳定控制策略。该模型在计算和硬件实现需求方面提供了有吸引力的权衡：无需外部内存缓冲或全局误差梯度计算，突触更新通过局部学习规则和广播的时序差分误差信号在线进行。因此，本研究有助于开发更硬件高效的强化学习解决方案。