Text-conditioned diffusion models have emerged as a promising tool for neural video generation. However, current models still struggle with intricate spatiotemporal prompts and often generate restricted or incorrect motion (e.g., even lacking the ability to be prompted for objects moving from left to right). To address these limitations, we introduce LLM-grounded Video Diffusion (LVD). Instead of directly generating videos from the text inputs, LVD first leverages a large language model (LLM) to generate dynamic scene layouts based on the text inputs and subsequently uses the generated layouts to guide a diffusion model for video generation. We show that LLMs are able to understand complex spatiotemporal dynamics from text alone and generate layouts that align closely with both the prompts and the object motion patterns typically observed in the real world. We then propose to guide video diffusion models with these layouts by adjusting the attention maps. Our approach is training-free and can be integrated into any video diffusion model that admits classifier guidance. Our results demonstrate that LVD significantly outperforms its base video diffusion model and several strong baseline methods in faithfully generating videos with the desired attributes and motion patterns.

翻译：基于文本条件的扩散模型已展现出用于神经视频生成的潜力。然而，当前模型在处理复杂的时空提示时仍存在困难，常常生成受限或错误的运动（例如，甚至无法生成物体从左到右移动的提示）。为解决这些局限性，我们提出了基于大语言模型的视频扩散方法（LLM-grounded Video Diffusion, LVD）。LVD并非直接从文本输入生成视频，而是首先利用大语言模型（LLM）根据文本输入生成动态场景布局，随后使用生成的布局引导扩散模型进行视频生成。我们证明，LLM能够仅从文本中理解复杂的时空动态，并生成与提示及现实世界中常见的物体运动模式高度一致的布局。接着，我们提出通过调整注意力图，利用这些布局引导视频扩散模型。该方法无需训练，可集成至任何支持分类器引导的视频扩散模型中。结果表明，LVD在忠实生成具有目标属性与运动模式的视频方面，显著优于其基础视频扩散模型及多个强基线方法。