Although actor-critic methods have been successful in practice, their theoretical analyses have several limitations. Specifically, existing theoretical work either sidesteps the exploration problem by making strong assumptions or analyzes impractical methods with complicated algorithmic modifications. Moreover, the actor-critic methods analyzed for linear MDPs often employ natural policy gradient and construct "implicit" policies without explicit parameterization. Such policies are computationally expensive to sample from, making the environment interactions inefficient. To that end, we focus on the finite-horizon linear MDPs and propose an optimistic actor-critic framework that uses parametric log-linear policies. In particular, we introduce a tractable $\textit{logit-matching}$ regression objective for the actor. For the critic, we use approximate Thompson sampling via Langevin Monte Carlo to obtain optimistic value estimates. We prove that the resulting algorithm achieves $\widetilde{\mathcal{O}}(ε^{-4})$ and $\widetilde{\mathcal{O}}(ε^{-2})$ sample complexity in the on-policy and off-policy setting, respectively. Our results match prior theoretical work in achieving the state-of-the-art sample complexity, while our algorithm is more aligned with practice.

翻译：尽管Actor-Critic方法在实践中取得了成功，但其理论分析仍存在若干局限性。具体而言，现有理论工作要么通过强假设回避探索问题，要么分析带有复杂算法修改的不可实际方法。此外，针对线性MDP所分析的Actor-Critic方法常采用自然策略梯度，并构造缺乏显式参数化的"隐式"策略。这类策略的样本采样计算成本高昂，导致环境交互效率低下。为此，我们聚焦于有限时域线性MDP，提出一种使用参数化对数线性策略的乐观Actor-Critic框架。特别地，我们为Actor引入了一个可解的$\textit{logit匹配}$回归目标。对于Critic，我们通过Langevin蒙特卡洛方法采用近似汤普森采样以获得乐观价值估计。我们证明，所提出的算法在同策略和异策略设置下分别达到$\widetilde{\mathcal{O}}(ε^{-4})$和$\widetilde{\mathcal{O}}(ε^{-2})$的样本复杂度。在实现最先进样本复杂度的同时，我们的结果与现有理论工作相当，而算法更贴近实际应用。