Neural Radiance Fields (NeRF) has received much attention recently due to its impressive capability to represent 3D scene and synthesize novel view images. Existing works usually assume that the input images are captured by a global shutter camera. Thus, rolling shutter (RS) images cannot be trivially applied to an off-the-shelf NeRF algorithm for novel view synthesis. Rolling shutter effect would also affect the accuracy of the camera pose estimation (e.g. via COLMAP), which further prevents the success of NeRF algorithm with RS images. In this paper, we propose Unrolling Shutter Bundle Adjusted Neural Radiance Fields (USB-NeRF). USB-NeRF is able to correct rolling shutter distortions and recover accurate camera motion trajectory simultaneously under the framework of NeRF, by modeling the physical image formation process of a RS camera. Experimental results demonstrate that USB-NeRF achieves better performance compared to prior works, in terms of RS effect removal, novel view image synthesis as well as camera motion estimation. Furthermore, our algorithm can also be used to recover high-fidelity high frame-rate global shutter video from a sequence of RS images.

翻译：神经辐射场（NeRF）近年来因其在三维场景表示和新视角图像合成方面的卓越能力而备受关注。现有工作通常假设输入图像由全局快门相机拍摄，因此滚动快门（RS）图像无法直接应用于现成的NeRF算法进行新视角合成。滚动快门效应还会影响相机位姿估计（例如通过COLMAP）的精度，进一步阻碍了NeRF算法在RS图像上的成功应用。本文提出滚动快门光束法平差神经辐射场（USB-NeRF）。USB-NeRF通过建模RS相机的物理成像过程，能够在NeRF框架下同时校正滚动快门畸变并恢复精确的相机运动轨迹。实验结果表明，与先前工作相比，USB-NeRF在RS效应去除、新视角图像合成以及相机运动估计方面均取得了更优性能。此外，本算法还可从RS图像序列中恢复高保真、高帧率的全局快门视频。