Recent implicit neural representations have shown great results for novel view synthesis. However, existing methods require expensive per-scene optimization from many views hence limiting their application to real-world unbounded urban settings where the objects of interest or backgrounds are observed from very few views. To mitigate this challenge, we introduce a new approach called NeO 360, Neural fields for sparse view synthesis of outdoor scenes. NeO 360 is a generalizable method that reconstructs 360{\deg} scenes from a single or a few posed RGB images. The essence of our approach is in capturing the distribution of complex real-world outdoor 3D scenes and using a hybrid image-conditional triplanar representation that can be queried from any world point. Our representation combines the best of both voxel-based and bird's-eye-view (BEV) representations and is more effective and expressive than each. NeO 360's representation allows us to learn from a large collection of unbounded 3D scenes while offering generalizability to new views and novel scenes from as few as a single image during inference. We demonstrate our approach on the proposed challenging 360{\deg} unbounded dataset, called NeRDS 360, and show that NeO 360 outperforms state-of-the-art generalizable methods for novel view synthesis while also offering editing and composition capabilities. Project page: https://zubair-irshad.github.io/projects/neo360.html

翻译：最近的隐式神经表示方法在新视角合成任务中取得了显著成果。然而，现有方法需要基于大量视角进行昂贵的逐场景优化，这限制了它们在现实世界无界城市环境中的应用——在这些环境中，目标物体或背景仅能从极少数视角观察到。针对这一挑战，我们提出了一种名为NeO 360的新方法，即面向户外场景稀疏视角合成的神经场（Neural fields for sparse view synthesis of outdoor scenes）。NeO 360是一种可泛化方法，能够通过单张或少量带位姿的RGB图像重建360°场景。该方法的核心在于捕捉复杂现实世界户外三维场景的分布，并采用一种混合图像条件三平面表示，该表示可从任意世界点进行查询。该表示融合了基于体素和鸟瞰图（BEV）两种表示的优点，比两者各自独立使用更为有效且更具表现力。NeO 360的表示使我们能够从大量无界三维场景集合中学习，同时在推理阶段仅需单张图像即可对新视角和新场景实现泛化。我们在所提出的具有挑战性的360°无界数据集NeRDS 360上展示了该方法，结果表明NeO 360在新视角合成方面优于当前最先进的可泛化方法，同时具备编辑和组合能力。项目页面：https://zubair-irshad.github.io/projects/neo360.html