Recent diffusion methods have made significant progress in generating videos from single images due to their powerful visual generation capabilities. However, challenges persist in image-to-video synthesis, particularly in human video generation, where inferring view-consistent, motion-dependent clothing wrinkles from a single image remains a formidable problem. In this paper, we present Human Video Generation in 4D (HVG), a latent video diffusion model capable of generating high-quality, multi-view, spatiotemporally coherent human videos from a single image with 3D pose and view control. HVG achieves this through three key designs: (i) Articulated Pose Modulation, which captures the anatomical relationships of 3D joints via a novel dual-dimensional bone map and resolves self-occlusions across views by introducing 3D information; (ii) View and Temporal Alignment, which ensures multi-view consistency and alignment between a reference image and pose sequences for frame-to-frame stability; and (iii) Progressive Spatio-Temporal Sampling with temporal alignment to maintain smooth transitions in long multi-view animations. Extensive experiments on image-to-video tasks demonstrate that HVG outperforms existing methods in generating high-quality 4D human videos from diverse human images and pose inputs.

翻译：近年来，扩散模型凭借其强大的视觉生成能力，在从单图像生成视频方面取得了显著进展。然而，图像到视频合成，特别是人体视频生成，仍然面临挑战。从单张图像推断出视角一致、运动相关的衣物褶皱，仍然是一个艰巨的问题。本文提出了四维人体视频生成模型，这是一种能够从单张图像，结合三维姿态和视角控制，生成高质量、多视角、时空一致的人体视频的潜在视频扩散模型。该模型通过三个关键设计实现这一目标：关节姿态调制，它通过一种新颖的双维度骨骼图捕捉三维关节的解剖学关系，并通过引入三维信息解决跨视角的自遮挡问题；视角与时间对齐，它确保多视角一致性以及参考图像与姿态序列之间的对齐，以实现帧间稳定性；以及结合时间对齐的渐进式时空采样，以保持长序列多视角动画的平滑过渡。在图像到视频任务上的大量实验表明，该模型在从多样化人体图像和姿态输入生成高质量四维人体视频方面，优于现有方法。