Despite the success of Random Network Distillation (RND) in various domains, it was shown as not discriminative enough to be used as an uncertainty estimator for penalizing out-of-distribution actions in offline reinforcement learning. In this paper, we revisit these results and show that, with a naive choice of conditioning for the RND prior, it becomes infeasible for the actor to effectively minimize the anti-exploration bonus and discriminativity is not an issue. We show that this limitation can be avoided with conditioning based on Feature-wise Linear Modulation (FiLM), resulting in a simple and efficient ensemble-free algorithm based on Soft Actor-Critic. We evaluate it on the D4RL benchmark, showing that it is capable of achieving performance comparable to ensemble-based methods and outperforming ensemble-free approaches by a wide margin.

翻译：尽管随机网络蒸馏（Random Network Distillation，RND）在多个领域取得了成功，但研究表明，在离线强化学习中，其区分度不足以作为不确定性估计器来惩罚分布外动作。本文重新审视了这些结果，并指出：若对RND先验采用朴素的调节方式，智能体将难以有效最小化反向探索奖励，且区分度并非问题所在。我们证明，基于特征线性调制（Feature-wise Linear Modulation，FiLM）的调节方法可以避免这一局限性，从而基于Soft Actor-Critic算法构建出简单高效的免集成方法。在D4RL基准测试上的实验表明，该方法能达到与基于集成的方法相媲美的性能，并以大幅度优势超越其他免集成方法。