The multi-agent setting is intricate and unpredictable since the behaviors of multiple agents influence one another. To address this environmental uncertainty, distributional reinforcement learning algorithms that incorporate uncertainty via distributional output have been integrated with multi-agent reinforcement learning (MARL) methods, achieving state-of-the-art performance. However, distributional MARL algorithms still rely on the traditional $\epsilon$-greedy, which does not take cooperative strategy into account. In this paper, we present a risk-based exploration that leads to collaboratively optimistic behavior by shifting the sampling region of distribution. Initially, we take expectations from the upper quantiles of state-action values for exploration, which are optimistic actions, and gradually shift the sampling region of quantiles to the full distribution for exploitation. By ensuring that each agent is exposed to the same level of risk, we can force them to take cooperatively optimistic actions. Our method shows remarkable performance in multi-agent settings requiring cooperative exploration based on quantile regression appropriately controlling the level of risk.

翻译：多智能体环境因多个智能体的行为相互影响而变得复杂且难以预测。为应对这种环境不确定性，通过分布输出引入不确定性的分布式强化学习算法已与多智能体强化学习（MARL）方法相结合，取得了最先进的性能。然而，分布式MARL算法仍依赖传统的$\epsilon$-贪心策略，该策略未考虑协作策略。本文提出一种基于风险的探索方法，通过移动分布的采样区域，引导合作性乐观行为。初始阶段，我们从状态-动作值的上分位数（即乐观动作）中提取期望值进行探索，并逐渐将分位数采样区域移动至完整分布以实现利用。通过确保每个智能体暴露于相同风险水平，可迫使它们采取合作性乐观行动。基于适当控制风险水平的分位数回归，该方法在需要协作探索的多智能体场景中展现出卓越性能。