The success of the Neural Radiance Fields (NeRFs) for modeling and free-view rendering static objects has inspired numerous attempts on dynamic scenes. Current techniques that utilize neural rendering for facilitating free-view videos (FVVs) are restricted to either offline rendering or are capable of processing only brief sequences with minimal motion. In this paper, we present a novel technique, Residual Radiance Field or ReRF, as a highly compact neural representation to achieve real-time FVV rendering on long-duration dynamic scenes. ReRF explicitly models the residual information between adjacent timestamps in the spatial-temporal feature space, with a global coordinate-based tiny MLP as the feature decoder. Specifically, ReRF employs a compact motion grid along with a residual feature grid to exploit inter-frame feature similarities. We show such a strategy can handle large motions without sacrificing quality. We further present a sequential training scheme to maintain the smoothness and the sparsity of the motion/residual grids. Based on ReRF, we design a special FVV codec that achieves three orders of magnitudes compression rate and provides a companion ReRF player to support online streaming of long-duration FVVs of dynamic scenes. Extensive experiments demonstrate the effectiveness of ReRF for compactly representing dynamic radiance fields, enabling an unprecedented free-viewpoint viewing experience in speed and quality.

翻译：神经辐射场（NeRFs）在建模和自由视角渲染静态物体方面的成功，激发了众多针对动态场景的研究尝试。当前利用神经渲染技术促进自由视角视频的技术，要么局限于离线渲染，要么仅能处理运动极小的短序列。本文提出一种新颖技术——残差辐射场（Residual Radiance Field, ReRF），作为一种高度紧凑的神经表示，能够对长时动态场景实现实时自由视角视频渲染。ReRF在时空特征空间中显式建模相邻时间戳之间的残差信息，并以基于全局坐标的小型MLP作为特征解码器。具体而言，ReRF采用紧凑的运动网格与残差特征网格，以利用帧间特征相似性。实验表明，该策略能够在保证质量的前提下处理大幅运动。我们进一步提出一种序列化训练方案，以保持运动/残差网格的光滑性与稀疏性。基于ReRF，我们设计了一种专用自由视角视频编解码器，实现了三个数量级的压缩率，并配套开发了ReRF播放器，支持动态场景长时自由视角视频的在线流式传输。大量实验验证了ReRF在紧凑表示动态辐射场方面的有效性，从而在速度和画质上实现了前所未有的自由视角观看体验。