Neural fields, a category of neural networks trained to represent high-frequency signals, have gained significant attention in recent years due to their impressive performance in modeling complex 3D data, such as signed distance (SDFs) or radiance fields (NeRFs), via a single multi-layer perceptron (MLP). However, despite the power and simplicity of representing signals with an MLP, these methods still face challenges when modeling large and complex temporal signals due to the limited capacity of MLPs. In this paper, we propose an effective approach to address this limitation by incorporating temporal residual layers into neural fields, dubbed ResFields. It is a novel class of networks specifically designed to effectively represent complex temporal signals. We conduct a comprehensive analysis of the properties of ResFields and propose a matrix factorization technique to reduce the number of trainable parameters and enhance generalization capabilities. Importantly, our formulation seamlessly integrates with existing MLP-based neural fields and consistently improves results across various challenging tasks: 2D video approximation, dynamic shape modeling via temporal SDFs, and dynamic NeRF reconstruction. Lastly, we demonstrate the practical utility of ResFields by showcasing its effectiveness in capturing dynamic 3D scenes from sparse RGBD cameras of a lightweight capture system.

翻译：神经场作为一种训练用于表示高频信号的神经网络类别，近年来因通过单一多层感知器（MLP）建模复杂三维数据（如符号距离函数或辐射场）的卓越性能而备受关注。然而，尽管使用MLP表示信号具有简洁高效的特点，但受限于MLP的容量限制，这些方法在建模大规模复杂时间信号时仍面临挑战。本文提出了一种有效方法——通过将时间残差层引入神经场（称为ResFields）来解决这一局限。这是一种专门用于高效表示复杂时间信号的新型网络架构。我们对ResFields的特性进行了全面分析，并提出了一种矩阵分解技术以减少可训练参数数量并增强泛化能力。重要的是，该公式可与基于MLP的现有神经场无缝集成，并在多种具有挑战性的任务中持续提升结果：二维视频近似、基于时间SDF的动态形状建模以及动态NeRF重建。最后，我们通过展示ResFields在轻量级捕捉系统中从稀疏RGBD相机捕捉动态三维场景的有效性，证明了其实用价值。