We study the variance of stochastic policy gradients (SPGs) with many action samples per state. We derive a many-actions optimality condition, which determines when many-actions SPG yields lower variance as compared to a single-action agent with proportionally extended trajectory. We propose Model-Based Many-Actions (MBMA), an approach leveraging dynamics models for many-actions sampling in the context of SPG. MBMA addresses issues associated with existing implementations of many-actions SPG and yields lower bias and comparable variance to SPG estimated from states in model-simulated rollouts. We find that MBMA bias and variance structure matches that predicted by theory. As a result, MBMA achieves improved sample efficiency and higher returns on a range of continuous action environments as compared to model-free, many-actions, and model-based on-policy SPG baselines.

翻译：我们研究了在每个状态下采用多个动作样本的随机策略梯度（SPG）的方差。我们推导了一个多动作最优条件，该条件决定了与按比例延长轨迹的单动作智能体相比，多动作SPG何时能产生更低的方差。我们提出了模型基多动作（MBMA）方法，这是一种在SPG背景下利用动力学模型进行多动作采样的技术。MBMA解决了现有多动作SPG实现中存在的问题，与从模型模拟轨迹中的状态估计的SPG相比，其偏差更低且方差相当。我们发现MBMA的偏差和方差结构与理论预测一致。因此，在一系列连续动作环境中，与无模型、多动作和基于模型的在线策略SPG基线相比，MBMA实现了更高的样本效率和回报。