As previous representations for reinforcement learning cannot effectively incorporate a human-intuitive understanding of the 3D environment, they usually suffer from sub-optimal performances. In this paper, we present Semantic-aware Neural Radiance Fields for Reinforcement Learning (SNeRL), which jointly optimizes semantic-aware neural radiance fields (NeRF) with a convolutional encoder to learn 3D-aware neural implicit representation from multi-view images. We introduce 3D semantic and distilled feature fields in parallel to the RGB radiance fields in NeRF to learn semantic and object-centric representation for reinforcement learning. SNeRL outperforms not only previous pixel-based representations but also recent 3D-aware representations both in model-free and model-based reinforcement learning.

翻译：先前的强化学习表示方法无法有效融入人类对三维环境的直观理解，因此往往面临性能次优的问题。本文提出面向强化学习的语义感知神经辐射场（SNeRL），该方法联合优化语义感知神经辐射场（NeRF）与卷积编码器，从多视角图像中学习三维感知的神经隐式表示。我们在NeRF的RGB辐射场中并行引入三维语义场与蒸馏特征场，为强化学习获取语义导向及以物体为中心的表示。实验表明，SNeRL在无模型与基于模型的强化学习中，不仅优于先前的像素级表示方法，也优于近期的三维感知表示方法。