We propose a new method for realistic real-time novel-view synthesis (NVS) of large scenes. Existing neural rendering methods generate realistic results, but primarily work for small scale scenes (<50 square meters) and have difficulty at large scale (>10000 square meters). Traditional graphics-based rasterization rendering is fast for large scenes but lacks realism and requires expensive manually created assets. Our approach combines the best of both worlds by taking a moderate-quality scaffold mesh as input and learning a neural texture field and shader to model view-dependant effects to enhance realism, while still using the standard graphics pipeline for real-time rendering. Our method outperforms existing neural rendering methods, providing at least 30x faster rendering with comparable or better realism for large self-driving and drone scenes. Our work is the first to enable real-time rendering of large real-world scenes.

翻译：我们提出了一种用于大场景逼真实时新视角合成的新方法。现有神经渲染方法能生成逼真结果，但主要适用于小规模场景（<50平方米），而在大规模场景（>10000平方米）上存在困难。传统的基于图形学光栅化渲染虽能快速处理大场景，但缺乏真实感且需要昂贵的手工创建资产。我们的方法结合了两者优势：以中等质量的骨架网格作为输入，学习神经纹理场和着色器来模拟视图相关效应以增强真实感，同时仍使用标准图形管线进行实时渲染。在自动驾驶和无人机大场景中，我们的方法优于现有神经渲染方法，渲染速度至少快30倍且真实感相当或更优。本工作是首个实现大规模真实场景实时渲染的方法。