The rising demand for creating lifelike avatars in the digital realm has led to an increased need for generating high-quality human videos guided by textual descriptions and poses. We propose Dancing Avatar, designed to fabricate human motion videos driven by poses and textual cues. Our approach employs a pretrained T2I diffusion model to generate each video frame in an autoregressive fashion. The crux of innovation lies in our adept utilization of the T2I diffusion model for producing video frames successively while preserving contextual relevance. We surmount the hurdles posed by maintaining human character and clothing consistency across varying poses, along with upholding the background's continuity amidst diverse human movements. To ensure consistent human appearances across the entire video, we devise an intra-frame alignment module. This module assimilates text-guided synthesized human character knowledge into the pretrained T2I diffusion model, synergizing insights from ChatGPT. For preserving background continuity, we put forth a background alignment pipeline, amalgamating insights from segment anything and image inpainting techniques. Furthermore, we propose an inter-frame alignment module that draws inspiration from an auto-regressive pipeline to augment temporal consistency between adjacent frames, where the preceding frame guides the synthesis process of the current frame. Comparisons with state-of-the-art methods demonstrate that Dancing Avatar exhibits the capacity to generate human videos with markedly superior quality, both in terms of human and background fidelity, as well as temporal coherence compared to existing state-of-the-art approaches.

翻译：随着数字领域对逼真化身需求的日益增长，基于文本描述与姿态引导的高质量人体视频生成技术愈发重要。本文提出“舞蹈化身”（Dancing Avatar）方法，旨在通过姿态与文本线索生成人体运动视频。该方法采用预训练的文本到图像（T2I）扩散模型，以自回归方式逐帧生成视频。核心创新在于巧妙利用T2I扩散模型逐帧生成图像的同时保持上下文相关性。我们克服了不同姿态下保持人物特征与服装一致性，以及多样化人体运动中维持背景连续性的挑战。为确保整段视频中人物外观的一致性，设计了帧内对齐模块：该模块将文本引导合成的人物知识融入预训练T2I扩散模型，并融合ChatGPT的语义信息。针对背景连续性，提出基于分割一切（SAM）与图像修复技术的背景对齐流程。此外，借鉴自回归流水线提出帧间对齐模块，通过前一帧引导当前帧合成过程，增强相邻帧的时间一致性。与现有最优方法的对比表明，舞蹈化身在人体与背景保真度以及时间连贯性方面，均能生成质量显著更优的人体视频。