Multi-agent planning and reinforcement learning can be challenging when agents cannot see the state of the world or communicate with each other due to communication costs, latency, or noise. Partially Observable Stochastic Games (POSGs) provide a mathematical framework for modelling such scenarios. This paper aims to improve the efficiency of planning and reinforcement learning algorithms for POSGs by identifying the underlying structure of optimal state-value functions. The approach involves reformulating the original game from the perspective of a trusted third party who plans on behalf of the agents simultaneously. From this viewpoint, the original POSGs can be viewed as Markov games where states are occupancy states, \ie posterior probability distributions over the hidden states of the world and the stream of actions and observations that agents have experienced so far. This study mainly proves that the optimal state-value function is a convex function of occupancy states expressed on an appropriate basis in all zero-sum, common-payoff, and Stackelberg POSGs.
翻译:多智能体规划与强化学习在智能体无法观测世界状态或因通信成本、延迟或噪声而无法相互通信时可能具有挑战性。部分可观测随机博弈(POSGs)为建模此类场景提供了数学框架。本文旨在通过识别最优状态值函数的潜在结构,提高针对POSGs的规划与强化学习算法的效率。该方法涉及从代表所有智能体同时进行规划的可信第三方角度重新构建原始博弈。从这一视角出发,原始POSGs可被视为马尔可夫博弈,其中状态即为占用状态,即关于世界隐藏状态以及智能体迄今为止经历的动作与观测流的后验概率分布。本研究主要证明,在所有零和、共同收益以及斯塔克尔伯格POSGs中,最优状态值函数是在适当基上表达的占用状态的凸函数。