We consider a distributed setup for reinforcement learning, where each agent has a copy of the same Markov Decision Process but transitions are sampled from the corresponding Markov chain independently by each agent. We show that in this setting, we can achieve a linear speedup for TD($\lambda$), a family of popular methods for policy evaluation, in the sense that $N$ agents can evaluate a policy $N$ times faster provided the target accuracy is small enough. Notably, this speedup is achieved by ``one shot averaging,'' a procedure where the agents run TD($\lambda$) with Markov sampling independently and only average their results after the final step. This significantly reduces the amount of communication required to achieve a linear speedup relative to previous work.

翻译：我们考虑强化学习中的分布式设置，其中每个智能体拥有相同的马尔可夫决策过程副本，但各智能体独立地从相应马尔可夫链中采样转移过程。研究表明在该设定下，对于策略评估领域流行的TD(λ)方法族，可实现线性加速效果——即当目标精度足够小时，N个智能体评估某个策略的速度可提升N倍。值得注意的是，这种加速通过"一次性平均"策略实现：各智能体独立运行马尔可夫采样的TD(λ)，仅在最终步骤后对结果进行平均。相较于先前工作，该方法显著降低了实现线性加速所需的通信量。