We study reinforcement learning with multinomial logistic (MNL) function approximation where the underlying transition probability kernel of the Markov decision processes (MDPs) is parametrized by an unknown transition core with features of state and action. For the finite horizon episodic setting with inhomogeneous state transitions, we propose provably efficient algorithms with randomized exploration having frequentist regret guarantees. For our first algorithm, $\texttt{RRL-MNL}$, we adapt optimistic sampling to ensure the optimism of the estimated value function with sufficient frequency and establish that $\texttt{RRL-MNL}$ is both statistically and computationally efficient, achieving a $\tilde{O}(\kappa^{-1} d^{\frac{3}{2}} H^{\frac{3}{2}} \sqrt{T})$ frequentist regret bound with constant-time computational cost per episode. Here, $d$ is the dimension of the transition core, $H$ is the horizon length, $T$ is the total number of steps, and $\kappa$ is a problem-dependent constant. Despite the simplicity and practicality of $\texttt{RRL-MNL}$, its regret bound scales with $\kappa^{-1}$, which is potentially large in the worst case. To improve the dependence on $\kappa^{-1}$, we propose $\texttt{ORRL-MNL}$, which estimates the value function using local gradient information of the MNL transition model. We show that its frequentist regret bound is $\tilde{O}(d^{\frac{3}{2}} H^{\frac{3}{2}} \sqrt{T} + \kappa^{-1} d^2 H^2)$. To the best of our knowledge, these are the first randomized RL algorithms for the MNL transition model that achieve both computational and statistical efficiency. Numerical experiments demonstrate the superior performance of the proposed algorithms.

翻译：我们研究基于多项式逻辑（MNL）函数逼近的强化学习，其中马尔可夫决策过程（MDP）的底层转移概率核由具有状态和动作特征的未知转移核参数化。针对具有非齐次状态转移的有限时间幕式设定，我们提出了具有频率主义遗憾保证的随机探索可证明高效算法。对于第一个算法 $\texttt{RRL-MNL}$，我们采用乐观采样以确保估计值函数以足够频率保持乐观性，并证明 $\texttt{RRL-MNL}$ 同时具有统计与计算效率，其频率主义遗憾界为 $\tilde{O}(\kappa^{-1} d^{\frac{3}{2}} H^{\frac{3}{2}} \sqrt{T})$，且每幕计算成本为常数时间。其中 $d$ 为转移核维度，$H$ 为时间跨度长度，$T$ 为总步数，$\kappa$ 为问题相关常数。尽管 $\texttt{RRL-MNL}$ 具有简洁性和实用性，其遗憾界与 $\kappa^{-1}$ 相关，在最坏情况下可能较大。为改善对 $\kappa^{-1}$ 的依赖，我们提出 $\texttt{ORRL-MNL}$，该算法利用 MNL 转移模型的局部梯度信息估计值函数。我们证明其频率主义遗憾界为 $\tilde{O}(d^{\frac{3}{2}} H^{\frac{3}{2}} \sqrt{T} + \kappa^{-1} d^2 H^2)$。据我们所知，这是首个针对 MNL 转移模型同时实现计算效率与统计效率的随机强化学习算法。数值实验证明了所提算法的优越性能。