Offline model selection (OMS), that is, choosing the best policy from a set of many policies given only logged data, is crucial for applying offline RL in real-world settings. One idea that has been extensively explored is to select policies based on the mean squared Bellman error (MSBE) of the associated Q-functions. However, previous work has struggled to obtain adequate OMS performance with Bellman errors, leading many researchers to abandon the idea. To this end, we elucidate why previous work has seen pessimistic results with Bellman errors and identify conditions under which OMS algorithms based on Bellman errors will perform well. Moreover, we develop a new estimator of the MSBE that is more accurate than prior methods. Our estimator obtains impressive OMS performance on diverse discrete control tasks, including Atari games.

翻译：离线模型选择（OMS）——即仅依据已有日志数据从一组策略中选出最优策略——对于在现实场景中应用离线强化学习至关重要。一个被广泛探索的思路是基于关联 Q 函数的均方贝尔曼误差（MSBE）来选择策略。然而，先前的研究在利用贝尔曼误差获得令人满意的 OMS 性能方面面临困难，导致许多研究者放弃这一思路。为此，我们阐释了先前研究在使用贝尔曼误差时得出悲观结果的原因，并确定了基于贝尔曼误差的 OMS 算法能够表现良好的条件。此外，我们开发了一种比先前方法更精确的 MSBE 新估计器。该估计器在包括 Atari 游戏在内的多种离散控制任务中取得了令人瞩目的 OMS 性能。