We present Kaleido, a family of generative models designed for photorealistic, unified object- and scene-level neural rendering. Kaleido operates on the principle that 3D can be regarded as a specialised sub-domain of video, expressed purely as a sequence-to-sequence image synthesis task. Through a systemic study of scaling sequence-to-sequence generative neural rendering, we introduce key architectural innovations that enable our model to: i) perform generative view synthesis without explicit 3D representations; ii) generate any number of 6-DoF target views conditioned on any number of reference views via a masked autoregressive framework; and iii) seamlessly unify 3D and video modelling within a single decoder-only rectified flow transformer. Within this unified framework, Kaleido leverages large-scale video data for pre-training, which significantly improves spatial consistency and reduces reliance on scarce, camera-labelled 3D datasets -- all without any architectural modifications. Kaleido sets a new state-of-the-art on a range of view synthesis benchmarks. Its zero-shot performance substantially outperforms other generative methods in few-view settings, and, for the first time, matches the quality of per-scene optimisation methods in many-view settings.

翻译：我们提出了Kaleido，一个旨在实现照片级真实感、统一对象与场景级神经渲染的生成模型系列。Kaleido基于一个核心原理运行：三维可被视为视频的一个特化子领域，纯粹表达为序列到序列的图像合成任务。通过对规模化序列到序列生成式神经渲染的系统性研究，我们引入了关键架构创新，使模型能够：i) 在没有显式三维表示的情况下执行生成式视角合成；ii) 通过掩码自回归框架，在任意数量参考视角的条件下生成任意数量的六自由度目标视角；iii) 在单一仅解码器的整流流Transformer中无缝统一三维与视频建模。在此统一框架内，Kaleido利用大规模视频数据进行预训练，显著提升了空间一致性并减少了对稀缺的相机标注三维数据集的依赖——所有这些都无需任何架构修改。Kaleido在一系列视角合成基准测试中创造了新的最优性能。其零样本性能在少视角设置下显著优于其他生成方法，并且首次在多视角设置下达到了逐场景优化方法的质量水平。