As previous representations for reinforcement learning cannot effectively incorporate a human-intuitive understanding of the 3D environment, they usually suffer from sub-optimal performances. In this paper, we present Semantic-aware Neural Radiance Fields for Reinforcement Learning (SNeRL), which jointly optimizes semantic-aware neural radiance fields (NeRF) with a convolutional encoder to learn 3D-aware neural implicit representation from multi-view images. We introduce 3D semantic and distilled feature fields in parallel to the RGB radiance fields in NeRF to learn semantic and object-centric representation for reinforcement learning. SNeRL outperforms not only previous pixel-based representations but also recent 3D-aware representations both in model-free and model-based reinforcement learning.

翻译：以往用于强化学习的表示方法无法有效融入对人类直观理解三维环境的能力，因此往往性能次优。本文提出面向强化学习的语义感知神经辐射场（SNeRL），该方法联合优化语义感知神经辐射场（NeRF）与卷积编码器，从多视角图像中学习三维感知的神经隐式表示。我们在NeRF的RGB辐射场基础上并行引入三维语义场与蒸馏特征场，为强化学习构建语义与物体中心表示。SNeRL不仅在无模型强化学习中超越以往基于像素的表示方法，且在有模型强化学习中亦胜过近期三维感知表示方法。