Multimodal large language models (MLLMs) and diffusion models have each reached remarkable maturity: MLLMs excel at reasoning over heterogeneous multimodal inputs with strong semantic grounding, while diffusion models synthesize images and videos with photorealistic fidelity. We argue that these two families can be unified through a simple division of labor: MLLMs perform semantic planning, while diffusion models render pixels from high-level semantic guidance and low-level visual features. Building on this idea, we propose Bernini, a unified framework for video generation and editing. An MLLM-based planner predicts the target semantic representation directly in the ViT embedding space, and a DiT-based renderer synthesizes pixels conditioned on this plan, augmented by text features and, for editing, source VAE features for detail preservation. Because semantics serve as the interface, the planner and renderer can be trained separately and only lightly co-trained, preserving the pretrained strengths of both components while keeping training efficient. To better handle multiple visual inputs, we introduce Segment-Aware 3D Rotary Positional Embedding (SA-3D RoPE), and further incorporate chain-of-thought reasoning in the planner to better transfer understanding into generation. Bernini achieves state-of-the-art performance across a wide range of video generation and editing benchmarks, with the MLLM's pretrained understanding translating into strong generalization on challenging editing tasks.

翻译：摘要：多模态大语言模型（MLLMs）与扩散模型均已达到高度成熟：MLLMs擅长在强语义根基下对异构多模态输入进行推理，而扩散模型则以逼真的真实感合成图像与视频。我们认为，这两类模型可通过简单的分工实现统一：MLLMs执行语义规划，扩散模型则根据高层语义引导与低层视觉特征渲染像素。基于这一思想，我们提出Bernini——一个统一的视频生成与编辑框架。基于MLLM的规划器直接在ViT嵌入空间预测目标语义表示，而基于DiT的渲染器则以此规划为条件合成像素，并结合文本特征（在编辑任务中还可利用源VAE特征保留细节）。由于语义作为交互界面，规划器与渲染器可分开训练，仅需少量联合训练，从而在保持高效训练的同时保留两个组件的预训练优势。为更好处理多路视觉输入，我们引入了感知分割的三维旋转位置编码（SA-3D RoPE），并在规划器中融入思维链推理，以更有效地将理解迁移至生成任务。Bernini在多项视频生成与编辑基准测试中达到了最先进性能，其MLLM的预训练理解能力成功转化为具有挑战性的编辑任务上的强泛化能力。