Modeling Neural Radiance Fields for fast-moving deformable objects from visual data alone is a challenging problem. A major issue arises due to the high deformation and low acquisition rates. To address this problem, we propose to use event cameras that offer very fast acquisition of visual change in an asynchronous manner. In this work, we develop a novel method to model the deformable neural radiance fields using RGB and event cameras. The proposed method uses the asynchronous stream of events and calibrated sparse RGB frames. In our setup, the camera pose at the individual events required to integrate them into the radiance fields remains unknown. Our method jointly optimizes these poses and the radiance field. This happens efficiently by leveraging the collection of events at once and actively sampling the events during learning. Experiments conducted on both realistically rendered graphics and real-world datasets demonstrate a significant benefit of the proposed method over the state-of-the-art and the compared baseline. This shows a promising direction for modeling deformable neural radiance fields in real-world dynamic scenes.

翻译：仅凭视觉数据对快速运动变形物体建模神经辐射场是一项具有挑战性的问题。主要困难源于高变形率和低采集速率。为解决该问题，我们提出使用能以异步方式快速采集视觉变化的事件相机。本文开发了一种利用RGB与事件相机建模可变形神经辐射场的新方法。该方法利用异步事件流与经标定的稀疏RGB帧。在我们的设置中，用于将单个事件集成到辐射场中的相机位姿仍属未知。本文方法联合优化这些位姿与辐射场，通过一次性利用事件集合并主动采样事件实现高效学习。在真实感渲染图形和真实世界数据集上的实验表明，所提方法相比现有最优技术和对比基线具有显著优势，为在真实动态场景中建模可变形神经辐射场指明了有前景的方向。