Despite the success of Random Network Distillation (RND) in various domains, it was shown as not discriminative enough to be used as an uncertainty estimator for penalizing out-of-distribution actions in offline reinforcement learning. In this paper, we revisit these results and show that, with a naive choice of conditioning for the RND prior, it becomes infeasible for the actor to effectively minimize the anti-exploration bonus and discriminativity is not an issue. We show that this limitation can be avoided with conditioning based on Feature-wise Linear Modulation (FiLM), resulting in a simple and efficient ensemble-free algorithm based on Soft Actor-Critic. We evaluate it on the D4RL benchmark, showing that it is capable of achieving performance comparable to ensemble-based methods and outperforming ensemble-free approaches by a wide margin.

翻译：尽管随机网络蒸馏（RND）在多个领域取得了成功，但研究表明，其作为不确定性估计器用于离线强化学习中惩罚分布外动作时，判别能力不足。本文重新审视了这些结论，并指出若对RND先验采用朴素的条件化方式，智能体将无法有效最小化反探索奖励，且判别能力并非问题根源。我们证明，基于特征线性调制（FiLM）的条件化可避免这一局限，从而提出一种基于Soft Actor-Critic的简单高效的无集成算法。在D4RL基准上的评估表明，该方法性能可与集成方法媲美，并以显著优势超越无集成方法。