Text-conditioned diffusion models have emerged as a promising tool for neural video generation. However, current models still struggle with intricate spatiotemporal prompts and often generate restricted or incorrect motion (e.g., even lacking the ability to be prompted for objects moving from left to right). To address these limitations, we introduce LLM-grounded Video Diffusion (LVD). Instead of directly generating videos from the text inputs, LVD first leverages a large language model (LLM) to generate dynamic scene layouts based on the text inputs and subsequently uses the generated layouts to guide a diffusion model for video generation. We show that LLMs are able to understand complex spatiotemporal dynamics from text alone and generate layouts that align closely with both the prompts and the object motion patterns typically observed in the real world. We then propose to guide video diffusion models with these layouts by adjusting the attention maps. Our approach is training-free and can be integrated into any video diffusion model that admits classifier guidance. Our results demonstrate that LVD significantly outperforms its base video diffusion model and several strong baseline methods in faithfully generating videos with the desired attributes and motion patterns.

翻译：文本条件扩散模型已成为神经视频生成领域中一种有前景的工具。然而，当前模型在处理复杂的时空提示时仍存在困难，常常生成受限或错误的运动（例如，甚至无法根据提示生成物体从左向右移动的场景）。为了克服这些局限，我们提出了LLM引导的视频扩散模型（LVD）。LVD并非直接从文本输入生成视频，而是首先利用大型语言模型（LLM）基于文本输入生成动态场景布局，随后使用生成的布局引导扩散模型进行视频生成。我们证明了LLM能够仅从文本中理解复杂的时空动态，并生成与提示以及真实世界中常见的物体运动模式高度一致的布局。接着，我们提出通过调整注意力图来利用这些布局引导视频扩散模型。我们的方法无需训练，可以集成到任何支持分类器引导的视频扩散模型中。实验结果表明，在忠实生成具有所需属性和运动模式的视频方面，LVD显著优于其基础视频扩散模型以及多种强有力的基线方法。