We introduce EscherNet, a multi-view conditioned diffusion model for view synthesis. EscherNet learns implicit and generative 3D representations coupled with a specialised camera positional encoding, allowing precise and continuous relative control of the camera transformation between an arbitrary number of reference and target views. EscherNet offers exceptional generality, flexibility, and scalability in view synthesis -- it can generate more than 100 consistent target views simultaneously on a single consumer-grade GPU, despite being trained with a fixed number of 3 reference views to 3 target views. As a result, EscherNet not only addresses zero-shot novel view synthesis, but also naturally unifies single- and multi-image 3D reconstruction, combining these diverse tasks into a single, cohesive framework. Our extensive experiments demonstrate that EscherNet achieves state-of-the-art performance in multiple benchmarks, even when compared to methods specifically tailored for each individual problem. This remarkable versatility opens up new directions for designing scalable neural architectures for 3D vision. Project page: https://kxhit.github.io/EscherNet.

翻译：我们提出EscherNet，一种多视角条件扩散模型，用于视图合成。EscherNet学习隐式且具有生成能力的3D表示，并结合专门的相机位置编码，从而实现对任意数量参考视图与目标视图之间相机变换的精确、连续相对控制。EscherNet在视图合成方面展现出卓越的通用性、灵活性与可扩展性——尽管仅基于固定数量的3个参考视图到3个目标视图进行训练，它仍能在单块消费级GPU上同时生成超过100个一致的目标视图。因此，EscherNet不仅解决了零样本新视图合成问题，还自然地统一了单图像与多图像的3D重建，将这些多样化的任务整合到一个统一的框架中。大量实验表明，即使与专门针对各自问题设计的方法相比，EscherNet在多个基准测试中仍达到了最先进的性能。这种卓越的通用性为设计面向3D视觉的可扩展神经架构开辟了新方向。项目页面：https://kxhit.github.io/EscherNet。