We study reinforcement learning (RL) with linear function approximation. For episodic time-inhomogeneous linear Markov decision processes (linear MDPs) whose transition probability can be parameterized as a linear function of a given feature mapping, we propose the first computationally efficient algorithm that achieves the nearly minimax optimal regret $\tilde O(d\sqrt{H^3K})$, where $d$ is the dimension of the feature mapping, $H$ is the planning horizon, and $K$ is the number of episodes. Our algorithm is based on a weighted linear regression scheme with a carefully designed weight, which depends on a new variance estimator that (1) directly estimates the variance of the optimal value function, (2) monotonically decreases with respect to the number of episodes to ensure a better estimation accuracy, and (3) uses a rare-switching policy to update the value function estimator to control the complexity of the estimated value function class. Our work provides a complete answer to optimal RL with linear MDPs, and the developed algorithm and theoretical tools may be of independent interest.

翻译：我们研究基于线性函数近似的强化学习（RL）。针对转移概率可参数化为给定特征映射线性函数的时变情节线性马尔可夫决策过程（线性MDP），我们提出首个计算高效的算法，实现了近极小极大最优遗憾界$\tilde O(d\sqrt{H^3K})$，其中$d$为特征映射维度，$H$为规划视界，$K$为情节数。该算法基于加权线性回归方案，采用精心设计的权重，该权重依赖于新的方差估计器，该估计器具备以下特性：(1) 直接估计最优值函数的方差；(2) 随情节数单调递减以确保更优的估计精度；(3) 采用稀疏切换策略更新值函数估计器以控制估计值函数类的复杂度。本研究完整回答了线性MDP的最优强化学习问题，所提出的算法与理论工具可能具有独立的学术价值。