Although reinforcement learning (RL) is considered the gold standard for policy design, it may not always provide a robust solution in various scenarios. This can result in severe performance degradation when the environment is exposed to potential disturbances. Adversarial training using a two-player max-min game has been proven effective in enhancing the robustness of RL agents. In this work, we extend the two-player game by introducing an adversarial herd, which involves a group of adversaries, in order to address ($\textit{i}$) the difficulty of the inner optimization problem, and ($\textit{ii}$) the potential over pessimism caused by the selection of a candidate adversary set that may include unlikely scenarios. We first prove that adversarial herds can efficiently approximate the inner optimization problem. Then we address the second issue by replacing the worst-case performance in the inner optimization with the average performance over the worst-$k$ adversaries. We evaluate the proposed method on multiple MuJoCo environments. Experimental results demonstrate that our approach consistently generates more robust policies.

翻译：尽管强化学习（RL）被视为策略设计的黄金标准，但在不同场景下未必总能提供鲁棒性解决方案。当环境面临潜在扰动时，这可能导致性能严重下降。采用双人最大-最小博弈的对抗训练已被证明能有效提升RL智能体的鲁棒性。本研究通过引入对抗群（即一组对抗者）扩展双人博弈，旨在解决：（$\textit{i}$）内层优化问题的求解困难，以及（$\textit{ii}$）因候选对抗集可能包含低概率场景而导致的过度悲观问题。我们首先证明对抗群能高效近似内层优化问题，随后通过将内层优化中的最差性能替换为最差-$k$个对抗者的平均性能来解决第二个问题。在多个MuJoCo环境上的实验评估表明，所提方法能持续生成更鲁棒的策略。