Motion-based controllable text-to-video generation involves motions to control the video generation. Previous methods typically require the training of models to encode motion cues or the fine-tuning of video diffusion models. However, these approaches often result in suboptimal motion generation when applied outside the trained domain. In this work, we propose MotionClone, a training-free framework that enables motion cloning from a reference video to control text-to-video generation. We employ temporal attention in video inversion to represent the motions in the reference video and introduce primary temporal-attention guidance to mitigate the influence of noisy or very subtle motions within the attention weights. Furthermore, to assist the generation model in synthesizing reasonable spatial relationships and enhance its prompt-following capability, we propose a location-aware semantic guidance mechanism that leverages the coarse location of the foreground from the reference video and original classifier-free guidance features to guide the video generation. Extensive experiments demonstrate that MotionClone exhibits proficiency in both global camera motion and local object motion, with notable superiority in terms of motion fidelity, textual alignment, and temporal consistency.

翻译：基于运动的可控文本到视频生成涉及使用运动来控制视频生成。先前的方法通常需要训练模型以编码运动线索，或对视频扩散模型进行微调。然而，这些方法在应用于训练域之外时，往往导致运动生成效果欠佳。在本工作中，我们提出了MotionClone，一个无需训练的框架，能够从参考视频克隆运动以控制文本到视频的生成。我们在视频反演中利用时序注意力来表示参考视频中的运动，并引入主时序注意力引导以减轻注意力权重中噪声或非常细微运动的影响。此外，为了辅助生成模型合成合理的空间关系并增强其提示跟随能力，我们提出了一种位置感知的语义引导机制，该机制利用参考视频中前景的粗略位置以及原始的无分类器引导特征来指导视频生成。大量实验表明，MotionClone在全局相机运动和局部物体运动方面均表现出色，在运动保真度、文本对齐度和时序一致性方面具有显著优势。