We present the OMG-CMDP! algorithm for regret minimization in adversarial Contextual MDPs. The algorithm operates under the minimal assumptions of realizable function class and access to online least squares and log loss regression oracles. Our algorithm is efficient (assuming efficient online regression oracles), simple and robust to approximation errors. It enjoys an $\widetilde{O}(H^{2.5} \sqrt{ T|S||A| ( \mathcal{R}(\mathcal{O}) + H \log(\delta^{-1}) )})$ regret guarantee, with $T$ being the number of episodes, $S$ the state space, $A$ the action space, $H$ the horizon and $\mathcal{R}(\mathcal{O}) = \mathcal{R}(\mathcal{O}_{\mathrm{sq}}^\mathcal{F}) + \mathcal{R}(\mathcal{O}_{\mathrm{log}}^\mathcal{P})$ is the sum of the regression oracles' regret, used to approximate the context-dependent rewards and dynamics, respectively. To the best of our knowledge, our algorithm is the first efficient rate optimal regret minimization algorithm for adversarial CMDPs that operates under the minimal standard assumption of online function approximation.

翻译：我们提出了OMG-CMDP!算法，用于在对抗性上下文MDP中实现遗憾最小化。该算法在可实现函数类以及可访问在线最小二乘和对数损失回归预言机的最小假设下运行。我们的算法高效（假设拥有高效的在线回归预言机）、简单且对近似误差具有鲁棒性。它享有$\widetilde{O}(H^{2.5} \sqrt{ T|S||A| ( \mathcal{R}(\mathcal{O}) + H \log(\delta^{-1}) )})$的遗憾保证，其中$T$为回合数，$S$为状态空间，$A$为动作空间，$H$为horizon，$\mathcal{R}(\mathcal{O}) = \mathcal{R}(\mathcal{O}_{\mathrm{sq}}^\mathcal{F}) + \mathcal{R}(\mathcal{O}_{\mathrm{log}}^\mathcal{P})$是用于分别近似上下文相关奖励和动态的回归预言机遗憾之和。据我们所知，我们的算法是首个在在线函数逼近这一最小标准假设下运行的对抗性CMDP高效率最优遗憾最小化算法。