Neural Radiance Fields (NeRF) has received much attention recently due to its impressive capability to represent 3D scene and synthesize novel view images. Existing works usually assume that the input images are captured by a global shutter camera. Thus, rolling shutter (RS) images cannot be trivially applied to an off-the-shelf NeRF algorithm for novel view synthesis. Rolling shutter effect would also affect the accuracy of the camera pose estimation (e.g. via COLMAP), which further prevents the success of NeRF algorithm with RS images. In this paper, we propose Unrolling Shutter Bundle Adjusted Neural Radiance Fields (USB-NeRF). USB-NeRF is able to correct rolling shutter distortions and recover accurate camera motion trajectory simultaneously under the framework of NeRF, by modeling the physical image formation process of a RS camera. Experimental results demonstrate that USB-NeRF achieves better performance compared to prior works, in terms of RS effect removal, novel view image synthesis as well as camera motion estimation. Furthermore, our algorithm can also be used to recover high-fidelity high frame-rate global shutter video from a sequence of RS images.

翻译：神经辐射场（NeRF）因其在三维场景表示和新视角图像合成方面的卓越能力，近年来受到广泛关注。现有工作通常假设输入图像由全局快门相机拍摄，因此滚动快门（RS）图像无法直接适用于现成的NeRF算法进行新视角合成。滚动快门效应还会影响相机位姿估计（如通过COLMAP）的精度，进一步阻碍NeRF算法在RS图像上的成功应用。本文提出滚动快门联合调整神经辐射场（USB-NeRF）。通过建模RS相机的物理成像过程，USB-NeRF能够在NeRF框架下同时校正滚动快门畸变并恢复准确的相机运动轨迹。实验结果表明，在RS效应消除、新视角图像合成以及相机运动估计方面，USB-NeRF的性能均优于现有方法。此外，本算法还可用于从一组RS图像中恢复高保真、高帧率的全局快门视频。