Novel view synthesis from a sparse set of input images is a challenging problem of great practical interest, especially when camera poses are absent or inaccurate. Direct optimization of camera poses and usage of estimated depths in neural radiance field algorithms usually do not produce good results because of the coupling between poses and depths, and inaccuracies in monocular depth estimation. In this paper, we leverage the recent 3D Gaussian splatting method to develop a novel construct-and-optimize method for sparse view synthesis without camera poses. Specifically, we construct a solution progressively by using monocular depth and projecting pixels back into the 3D world. During construction, we optimize the solution by detecting 2D correspondences between training views and the corresponding rendered images. We develop a unified differentiable pipeline for camera registration and adjustment of both camera poses and depths, followed by back-projection. We also introduce a novel notion of an expected surface in Gaussian splatting, which is critical to our optimization. These steps enable a coarse solution, which can then be low-pass filtered and refined using standard optimization methods. We demonstrate results on the Tanks and Temples and Static Hikes datasets with as few as three widely-spaced views, showing significantly better quality than competing methods, including those with approximate camera pose information. Moreover, our results improve with more views and outperform previous InstantNGP and Gaussian Splatting algorithms even when using half the dataset.

翻译：从稀疏输入图像进行新颖视图合成是一个具有实际挑战性的问题，尤其在相机姿态缺失或不准确时。由于姿态与深度的耦合以及单目深度估计的不准确性，直接优化相机姿态或在神经辐射场算法中使用估计深度通常难以取得良好效果。本文利用近期提出的3D高斯泼溅方法，开发了一种新颖的"构造-优化"方法，用于在不依赖相机姿态的情况下实现稀疏视图合成。具体而言，我们通过单目深度将像素反投影至三维世界，逐步构建解决方案。在构建过程中，通过检测训练视图与对应渲染图像之间的二维对应关系来优化解。我们开发了一个统一的微分管道，用于相机配准以及相机姿态与深度的联合调整，随后执行反投影操作。我们还引入了高斯泼溅中期望表面的新概念，这对优化至关重要。这些步骤可生成粗解，随后通过低通滤波并使用标准优化方法进行精炼。我们在Tanks and Temples和Static Hikes数据集上仅使用三个宽基线视图展示了结果，质量显著优于竞争方法（包括那些具备近似相机姿态信息的方法）。此外，我们的结果随视图数量增加而提升，且在使用半数数据集的情况下仍优于先前InstantNGP和高斯泼溅算法。