Camera-controlled generative video re-rendering methods, such as ReCamMaster, have achieved remarkable progress. However, despite their success in single-view setting, these works often struggle to maintain consistency across multi-view scenarios. Ensuring spatio-temporal coherence in hallucinated regions remains challenging due to the inherent stochasticity of generative models. To address it, we introduce PlenopticDreamer, a framework that synchronizes generative hallucinations to maintain spatio-temporal memory. The core idea is to train a multi-in-single-out video-conditioned model in an autoregressive manner, aided by a camera-guided video retrieval strategy that adaptively selects salient videos from previous generations as conditional inputs. In addition, Our training incorporates progressive context-scaling to improve convergence, self-conditioning to enhance robustness against long-range visual degradation caused by error accumulation, and a long-video conditioning mechanism to support extended video generation. Extensive experiments on the Basic and Agibot benchmarks demonstrate that PlenopticDreamer achieves state-of-the-art video re-rendering, delivering superior view synchronization, high-fidelity visuals, accurate camera control, and diverse view transformations (e.g., third-person to third-person, and head-view to gripper-view in robotic manipulation). Project page: https://research.nvidia.com/labs/dir/plenopticdreamer/

翻译：相机控制的生成式视频重渲染方法（如ReCamMaster）已取得显著进展。然而，尽管在单视图场景中取得成功，这些方法在多视图场景中往往难以保持一致性。由于生成模型固有的随机性，确保幻觉区域的时空一致性仍然具有挑战性。为此，我们提出了PlenopticDreamer框架，该框架通过同步生成幻觉来维持时空记忆。其核心思想是以自回归方式训练一个多输入单输出的视频条件模型，并辅以相机引导的视频检索策略，该策略自适应地从先前生成过程中选择显著视频作为条件输入。此外，我们的训练结合了渐进式上下文扩展以改善收敛性，自条件机制以增强对误差累积导致的长期视觉退化的鲁棒性，以及长视频条件机制以支持扩展视频生成。在Basic和Agibot基准上的大量实验表明，PlenopticDreamer实现了最先进的视频重渲染效果，提供了卓越的视图同步性、高保真视觉效果、精确的相机控制以及多样化的视图变换（例如，从第三人称到第三人称，以及机器人操作中从头部视角到夹爪视角的转换）。项目页面：https://research.nvidia.com/labs/dir/plenopticdreamer/