As previous representations for reinforcement learning cannot effectively incorporate a human-intuitive understanding of the 3D environment, they usually suffer from sub-optimal performances. In this paper, we present Semantic-aware Neural Radiance Fields for Reinforcement Learning (SNeRL), which jointly optimizes semantic-aware neural radiance fields (NeRF) with a convolutional encoder to learn 3D-aware neural implicit representation from multi-view images. We introduce 3D semantic and distilled feature fields in parallel to the RGB radiance fields in NeRF to learn semantic and object-centric representation for reinforcement learning. SNeRL outperforms not only previous pixel-based representations but also recent 3D-aware representations both in model-free and model-based reinforcement learning.

翻译：先前的强化学习表示方法无法有效融入人类对三维环境的直观理解，因此通常难以达到最优性能。本文提出面向强化学习的语义感知神经辐射场（SNeRL），该方法通过联合优化语义感知神经辐射场（NeRF）与卷积编码器，从多视角图像中学习三维感知的神经隐式表示。我们在NeRF的RGB辐射场基础上并行引入三维语义场与蒸馏特征场，为强化学习构建语义与面向对象的表征。SNeRL不仅在无模型与基于模型的强化学习中超越先前基于像素的表示方法，其性能也优于近期提出的三维感知表示方法。