We introduce EscherNet, a multi-view conditioned diffusion model for view synthesis. EscherNet learns implicit and generative 3D representations coupled with a specialised camera positional encoding, allowing precise and continuous relative control of the camera transformation between an arbitrary number of reference and target views. EscherNet offers exceptional generality, flexibility, and scalability in view synthesis -- it can generate more than 100 consistent target views simultaneously on a single consumer-grade GPU, despite being trained with a fixed number of 3 reference views to 3 target views. As a result, EscherNet not only addresses zero-shot novel view synthesis, but also naturally unifies single- and multi-image 3D reconstruction, combining these diverse tasks into a single, cohesive framework. Our extensive experiments demonstrate that EscherNet achieves state-of-the-art performance in multiple benchmarks, even when compared to methods specifically tailored for each individual problem. This remarkable versatility opens up new directions for designing scalable neural architectures for 3D vision. Project page: \url{https://kxhit.github.io/EscherNet}.

翻译：我们提出EscherNet，一种用于视图合成的多视角条件扩散模型。EscherNet学习隐式且具有生成能力的3D表示，并结合专用的相机位置编码，从而在任意数量的参考视图与目标视图之间实现对相机变换的精确且连续的相对控制。EscherNet在视图合成中展现出非凡的通用性、灵活性和可扩展性——尽管训练时仅使用固定的3张参考视图生成3张目标视图，它能在单个消费级GPU上同时生成超过100张一致的目标视图。因此，EscherNet不仅能够处理零样本新视图合成，还能自然地统一单幅图像和多幅图像的3D重建，将这些多样化的任务融合为一个连贯的框架。我们的广泛实验表明，即使在针对每个具体问题专门设计的方法中，EscherNet也在多个基准测试中达到了最先进的性能。这种卓越的多功能性为设计用于3D视觉的可扩展神经架构开辟了新方向。项目页面：\url{https://kxhit.github.io/EscherNet}。