We introduce an improved solution to the neural image-based rendering problem in computer vision. Given a set of images taken from a freely moving camera at train time, the proposed approach could synthesize a realistic image of the scene from a novel viewpoint at test time. The key ideas presented in this paper are (i) Recovering accurate camera parameters via a robust pipeline from unposed day-to-day images is equally crucial in neural novel view synthesis problem; (ii) It is rather more practical to model object's content at different resolutions since dramatic camera motion is highly likely in day-to-day unposed images. To incorporate the key ideas, we leverage the fundamentals of scene rigidity, multi-scale neural scene representation, and single-image depth prediction. Concretely, the proposed approach makes the camera parameters as learnable in a neural fields-based modeling framework. By assuming per view depth prediction is given up to scale, we constrain the relative pose between successive frames. From the relative poses, absolute camera pose estimation is modeled via a graph-neural network-based multiple motion averaging within the multi-scale neural-fields network, leading to a single loss function. Optimizing the introduced loss function provides camera intrinsic, extrinsic, and image rendering from unposed images. We demonstrate, with examples, that for a unified framework to accurately model multiscale neural scene representation from day-to-day acquired unposed multi-view images, it is equally essential to have precise camera-pose estimates within the scene representation framework. Without considering robustness measures in the camera pose estimation pipeline, modeling for multi-scale aliasing artifacts can be counterproductive. We present extensive experiments on several benchmark datasets to demonstrate the suitability of our approach.

翻译：我们提出了一种改进的解决方案，用于计算机视觉中的神经图像渲染问题。给定一组由自由移动相机在训练时拍摄的图像，所提出的方法能够在测试时从新视角合成场景的真实感图像。本文的核心思想包括：(i) 从非位姿的日常图像中通过鲁棒流程恢复精确的相机参数，在神经新视角合成问题中同样至关重要；(ii) 由于日常非位姿图像中很可能出现剧烈的相机运动，以不同分辨率建模物体内容更为实用。为融合这些思想，我们利用了场景刚性、多尺度神经场景表示以及单图像深度预测的基础原理。具体而言，所提出的方法将相机参数设为神经场建模框架中的可学习参数。通过假设每视角的深度预测在尺度上已知，我们约束了连续帧之间的相对位姿。基于相对位姿，通过图神经网络的多重运动平均方法，在多尺度神经场网络框架内建模绝对相机位姿估计，从而得到单一损失函数。优化该损失函数可从非位姿图像中获取相机内参、外参及图像渲染结果。我们通过示例证明：若要构建统一框架，从日常采集的非位姿多视角图像中精确建模多尺度神经场景表示，那么在该场景表示框架内获得精确的相机位姿估计同样不可或缺。若未在相机位姿估计流程中考虑鲁棒性措施，多尺度混叠伪影的建模可能适得其反。我们在多个基准数据集上进行了大量实验，以证明我们方法的适用性。