Generating text-editable and pose-controllable character videos have an imperious demand in creating various digital human. Nevertheless, this task has been restricted by the absence of a comprehensive dataset featuring paired video-pose captions and the generative prior models for videos. In this work, we design a novel two-stage training scheme that can utilize easily obtained datasets (i.e.,image pose pair and pose-free video) and the pre-trained text-to-image (T2I) model to obtain the pose-controllable character videos. Specifically, in the first stage, only the keypoint-image pairs are used only for a controllable text-to-image generation. We learn a zero-initialized convolutional encoder to encode the pose information. In the second stage, we finetune the motion of the above network via a pose-free video dataset by adding the learnable temporal self-attention and reformed cross-frame self-attention blocks. Powered by our new designs, our method successfully generates continuously pose-controllable character videos while keeps the editing and concept composition ability of the pre-trained T2I model. The code and models will be made publicly available.

翻译：生成可文本编辑且姿势可控的角色视频在创建多种数字人方面具有迫切需求。然而，由于缺乏包含配对视频-姿态标注的综合数据集以及视频生成先验模型，该任务一直受到限制。在本工作中，我们设计了一种新颖的两阶段训练方案，能够利用易于获取的数据集（即图像-姿态对和无姿态视频）以及预训练的文本到图像（T2I）模型来生成姿势可控的角色视频。具体而言，在第一阶段，仅使用关键点-图像对用于可控的文本到图像生成。我们学习一个零初始化的卷积编码器来编码姿态信息。在第二阶段，我们通过添加可学习的时序自注意力模块和重构的跨帧自注意力模块，在无姿态视频数据集上微调上述网络的运动部分。得益于我们的新设计，该方法成功生成了连续姿势可控的角色视频，同时保留了预训练T2I模型的编辑和概念组合能力。相关代码和模型将公开提供。