Modern learning-based approaches to 3D-aware image synthesis achieve high photorealism and 3D-consistent viewpoint changes for the generated images. Existing approaches represent instances in a shared canonical space. However, for in-the-wild datasets a shared canonical system can be difficult to define or might not even exist. In this work, we instead model instances in view space, alleviating the need for posed images and learned camera distributions. We find that in this setting, existing GAN-based methods are prone to generating flat geometry and struggle with distribution coverage. We hence propose WildFusion, a new approach to 3D-aware image synthesis based on latent diffusion models (LDMs). We first train an autoencoder that infers a compressed latent representation, which additionally captures the images' underlying 3D structure and enables not only reconstruction but also novel view synthesis. To learn a faithful 3D representation, we leverage cues from monocular depth prediction. Then, we train a diffusion model in the 3D-aware latent space, thereby enabling synthesis of high-quality 3D-consistent image samples, outperforming recent state-of-the-art GAN-based methods. Importantly, our 3D-aware LDM is trained without any direct supervision from multiview images or 3D geometry and does not require posed images or learned pose or camera distributions. It directly learns a 3D representation without relying on canonical camera coordinates. This opens up promising research avenues for scalable 3D-aware image synthesis and 3D content creation from in-the-wild image data. See https://katjaschwarz.github.io/wildfusion for videos of our 3D results.

翻译：现代基于学习的三维感知图像合成方法在生成图像上实现了高逼真度与三维一致的视角变换。现有方法将实例表征在共享的规范空间中，然而对于非受控数据集，规范系统往往难以定义甚至根本不存在。本文转而将实例建模于视角空间中，从而消除了对带姿态图像及学习相机分布的需求。我们发现，在此设定下，现有基于GAN的方法易产生扁平几何结构且难以覆盖数据分布。为此我们提出WildFusion——一种基于潜扩散模型的三维感知图像合成新方法。首先训练自编码器推断压缩潜表征，该表征不仅捕获图像底层三维结构，更支持重建与新颖视角合成。为学习可信三维表征，我们利用单目深度预测线索。随后在三维感知潜空间中训练扩散模型，从而合成高质量三维一致图像样本，性能超越当前最优的GAN方法。关键的是，我们的三维感知LDM无需多视图图像或三维几何的直接监督，也不依赖带姿态图像、预设姿态或相机分布，直接在无规范相机坐标的条件下学习三维表征。这为基于非受控图像数据的可扩展三维感知图像合成与三维内容创作开辟了前景广阔的研究方向。三维结果视频参见https://katjaschwarz.github.io/wildfusion。