Self-supervised novel view synthesis (NVS) remains challenging to scale, despite the abundance of video data, largely due to the brittleness of training on realistic videos and the hard-to-predict scaling behavior of multi-network system designs. We introduce RayDer, a unified, feed-forward transformer that consolidates camera estimation, scene reconstruction, and rendering into a single backbone, turning self-supervised NVS into a well-posed single-model scaling problem. A minimal dynamic state, treated as a nuisance factor, absorbs time-varying content and enables stable training on unconstrained real-world video. Importantly, RayDer keeps static-scene NVS as its target task: dynamic content is leveraged purely as scalable supervision, not reconstructed as in dynamic-scene (4D) NVS. Across multiple model sizes and orders of magnitude in data, RayDer exhibits clean power-law scaling with data and compute, and outperforms static-scene data mixtures. On a large number of benchmarks, RayDer achieves strong zero-shot open-set performance competitive with state-of-the-art supervised approaches. Project Page: https://compvis.github.io/rayder

翻译：自监督新视角合成（NVS）在大规模应用中仍面临挑战，尽管视频数据丰富，这主要源于在真实视频上训练的脆弱性以及多网络系统设计难以预测的扩展行为。我们提出RayDer，一种统一的前馈式Transformer架构，将相机估计、场景重建与渲染整合至单一主干网络，使自监督NVS转化为一个良态的单模型扩展问题。通过将最小动态状态视为干扰因子，该模型吸收时变内容，实现对无约束真实世界视频的稳定训练。关键在于，RayDer始终以静态场景NVS为目标任务：动态内容仅作为可扩展的监督信号被利用，而非如动态场景（4D）NVS那样进行重建。在多种模型规模与跨数量级数据量下，RayDer展现出与数据和计算量呈清晰幂律扩展的特性，并优于静态场景数据混合方案。在大量基准测试中，RayDer在零样本开放场景下取得了与顶尖监督方法相匹敌的强劲性能。项目主页：https://compvis.github.io/rayder