A practical challenge in reinforcement learning are combinatorial action spaces that make planning computationally demanding. For example, in cooperative multi-agent reinforcement learning, a potentially large number of agents jointly optimize a global reward function, which leads to a combinatorial blow-up in the action space by the number of agents. As a minimal requirement, we assume access to an argmax oracle that allows to efficiently compute the greedy policy for any Q-function in the model class. Building on recent work in planning with local access to a simulator and linear function approximation, we propose efficient algorithms for this setting that lead to polynomial compute and query complexity in all relevant problem parameters. For the special case where the feature decomposition is additive, we further improve the bounds and extend the results to the kernelized setting with an efficient algorithm.

翻译：强化学习中的一个实际挑战是组合动作空间，这导致规划计算复杂度高。例如，在合作多智能体强化学习中，大量智能体共同优化一个全局奖励函数，使得动作空间随智能体数量呈组合爆炸式增长。作为最低要求，我们假设能够访问一个argmax预言机，从而可以高效计算模型类中任意Q函数的贪心策略。基于近期在局部访问模拟器与线性函数近似下的规划研究成果，我们针对该场景提出了高效算法，该算法在所有相关问题参数上均具有多项式计算复杂度和查询复杂度。对于特征分解为可加形式的特殊情况，我们进一步改进了算法边界，并将结果推广到核化设定下，同时提出了相应的有效算法。