Text-to-video models have made significant strides in generating short video clips from textual descriptions. Yet, a significant challenge remains: generating several video shots of the same characters, preserving their identity without hurting video quality, dynamics, and responsiveness to text prompts. We present Video Storyboarding, a training-free method to enable pretrained text-to-video models to generate multiple shots with consistent characters, by sharing features between them. Our key insight is that self-attention query features (Q) encode both motion and identity. This creates a hard-to-avoid trade-off between preserving character identity and making videos dynamic, when features are shared. To address this issue, we introduce a novel query injection strategy that balances identity preservation and natural motion retention. This approach improves upon naive consistency techniques applied to videos, which often struggle to maintain this delicate equilibrium. Our experiments demonstrate significant improvements in character consistency across scenes while maintaining high-quality motion and text alignment. These results offer insights into critical stages of video generation and the interplay of structure and motion in video diffusion models.

翻译：文本到视频模型在根据文本描述生成短视频片段方面取得了显著进展。然而，一个重大挑战依然存在：生成同一角色的多个视频镜头，在保持角色身份的同时不损害视频质量、动态性以及对文本提示的响应能力。我们提出了视频故事板技术，这是一种无需训练的方法，能够通过共享特征使预训练的文本到视频模型生成具有一致角色的多个镜头。我们的关键见解是，自注意力查询特征（Q）同时编码了运动和身份信息。当特征被共享时，这导致在保持角色身份和实现视频动态性之间难以避免的权衡。为了解决这一问题，我们引入了一种新颖的查询注入策略，以平衡身份保持和自然运动保留。这种方法改进了应用于视频的简单一致性技术，后者往往难以维持这种微妙的平衡。我们的实验表明，在保持高质量运动和文本对齐的同时，跨场景的角色一致性得到了显著改善。这些结果为视频生成的关键阶段以及视频扩散模型中结构与运动的相互作用提供了深入见解。