Modeling Neural Radiance Fields for fast-moving deformable objects from visual data alone is a challenging problem. A major issue arises due to the high deformation and low acquisition rates. To address this problem, we propose to use event cameras that offer very fast acquisition of visual change in an asynchronous manner. In this work, we develop a novel method to model the deformable neural radiance fields using RGB and event cameras. The proposed method uses the asynchronous stream of events and calibrated sparse RGB frames. In our setup, the camera pose at the individual events required to integrate them into the radiance fields remains unknown. Our method jointly optimizes these poses and the radiance field. This happens efficiently by leveraging the collection of events at once and actively sampling the events during learning. Experiments conducted on both realistically rendered graphics and real-world datasets demonstrate a significant benefit of the proposed method over the state-of-the-art and the compared baseline. This shows a promising direction for modeling deformable neural radiance fields in real-world dynamic scenes.

翻译：仅从视觉数据对快速运动可变形物体进行神经辐射场建模是一个具有挑战性的问题。主要难点在于物体高度形变与低采集速率之间的矛盾。针对该问题，我们提出采用能以异步方式高速采集视觉变化的事件相机。本文提出了一种利用RGB相机与事件相机对可变形神经辐射场进行建模的新方法。该方法综合利用异步事件流与经过标定的稀疏RGB图像帧。在我们的设置中，用于将单个事件集成到辐射场中的相机位姿是未知的。本文方法通过联合优化这些位姿与辐射场来实现高效建模，其关键在于一次性利用所有事件数据并在学习过程中主动采样事件。在逼真渲染图形与真实世界数据集上的实验表明，所提方法相较于现有最新技术及对比基线具有显著优势，为真实动态场景中可变形神经辐射场建模指明了有前景的研究方向。