Neural fields, a category of neural networks trained to represent high-frequency signals, have gained significant attention in recent years due to their impressive performance in modeling complex 3D data, such as signed distance (SDFs) or radiance fields (NeRFs), via a single multi-layer perceptron (MLP). However, despite the power and simplicity of representing signals with an MLP, these methods still face challenges when modeling large and complex temporal signals due to the limited capacity of MLPs. In this paper, we propose an effective approach to address this limitation by incorporating temporal residual layers into neural fields, dubbed ResFields. It is a novel class of networks specifically designed to effectively represent complex temporal signals. We conduct a comprehensive analysis of the properties of ResFields and propose a matrix factorization technique to reduce the number of trainable parameters and enhance generalization capabilities. Importantly, our formulation seamlessly integrates with existing MLP-based neural fields and consistently improves results across various challenging tasks: 2D video approximation, dynamic shape modeling via temporal SDFs, and dynamic NeRF reconstruction. Lastly, we demonstrate the practical utility of ResFields by showcasing its effectiveness in capturing dynamic 3D scenes from sparse RGBD cameras of a lightweight capture system.

翻译：神经场是一类用于表示高频信号的神经网络，近年来因其通过单层多层感知机（MLP）在建模复杂三维数据（如符号距离函数（SDF）或辐射场（NeRF））方面的卓越表现而备受关注。然而，尽管利用MLP表示信号具有强大且简洁的优势，但由于MLP容量的限制，这些方法在建模大规模复杂时间信号时仍面临挑战。本文提出了一种有效方法来解决这一局限性：将时间残差层融入神经场，称为ResFields。这是一种专为有效表示复杂时间信号而设计的新型网络。我们对ResFields的性质进行了全面分析，并提出了一种矩阵分解技术，以减少可训练参数数量并提升泛化能力。重要的是，我们的公式能无缝集成至现有基于MLP的神经场，并在多项具有挑战性的任务中持续提升结果：二维视频逼近、基于时间SDF的动态形状建模以及动态NeRF重建。最后，我们通过展示ResFields在轻量级捕获系统中从稀疏RGBD相机捕捉动态三维场景的有效性，验证了其实用价值。