Text-guided generative diffusion models unlock powerful image creation and editing tools. While these have been extended to video generation, current approaches that edit the content of existing footage while retaining structure require expensive re-training for every input or rely on error-prone propagation of image edits across frames. In this work, we present a structure and content-guided video diffusion model that edits videos based on visual or textual descriptions of the desired output. Conflicts between user-provided content edits and structure representations occur due to insufficient disentanglement between the two aspects. As a solution, we show that training on monocular depth estimates with varying levels of detail provides control over structure and content fidelity. Our model is trained jointly on images and videos which also exposes explicit control of temporal consistency through a novel guidance method. Our experiments demonstrate a wide variety of successes; fine-grained control over output characteristics, customization based on a few reference images, and a strong user preference towards results by our model.

翻译：文本引导的扩散模型为图像生成与编辑提供了强大工具。尽管该类模型已扩展至视频生成领域，但现有方法在保留原始视频结构的同时编辑内容时，要么需要针对每个输入进行昂贵的重训练，要么依赖逐帧图像编辑中易出错的传播机制。本研究提出一种结构及内容引导的视频扩散模型，可基于用户对预期输出的视觉或文字描述进行视频编辑。由于结构表示与内容编辑之间的解耦不充分，用户输入的内容修改与结构表征之间会产生冲突。为解决该问题，我们证明采用不同细节层次单目深度估计进行训练，能够实现对结构保真度与内容保真度的可控调节。该模型在图像与视频联合数据上训练，并借助新型引导方法实现了对时间一致性的显式控制。实验表明，本模型在输出特征细粒度控制、基于少量参考图像的个性化定制以及用户偏好评估方面均展现出显著优势。