We concentrate on a novel human-centric image synthesis task, that is, given only one reference facial photograph, it is expected to generate specific individual images with diverse head positions, poses, and facial expressions in different contexts. To accomplish this goal, we argue that our generative model should be capable of the following favorable characteristics: (1) a strong visual and semantic understanding of our world and human society for basic object and human image generation. (2) generalizable identity preservation ability. (3) flexible and fine-grained head control. Recently, large pre-trained text-to-image diffusion models have shown remarkable results, serving as a powerful generative foundation. As a basis, we aim to unleash the above two capabilities of the pre-trained model. In this work, we present a new framework named CapHuman. We embrace the ``encode then learn to align" paradigm, which enables generalizable identity preservation for new individuals without cumbersome tuning at inference. CapHuman encodes identity features and then learns to align them into the latent space. Moreover, we introduce the 3D facial prior to equip our model with control over the human head in a flexible and 3D-consistent manner. Extensive qualitative and quantitative analyses demonstrate our CapHuman can produce well-identity-preserved, photo-realistic, and high-fidelity portraits with content-rich representations and various head renditions, superior to established baselines. Code and checkpoint will be released at https://github.com/VamosC/CapHuman.

翻译：我们聚焦于一项新颖的以人为中心的图像合成任务，即仅需给定一张参考面部照片，即可生成特定个体在不同场景下具有多样化头部位置、姿态和面部表情的图像。为实现该目标，我们认为生成模型应具备以下理想特性：（1）对世界和人类社会具备强大的视觉与语义理解能力，以实现基础物体和人物图像生成；（2）具有泛化的身份保持能力；（3）灵活且细粒度的头部控制能力。近期，大规模预训练的文本到图像扩散模型已展现出显著效果，可作为强大的生成基础。基于此，我们旨在释放预训练模型的上述两种能力。本文提出名为CapHuman的新框架，采用“编码后学习对齐”范式，无需在推理时进行繁琐调优即可实现对新个体的泛化身份保持。CapHuman编码身份特征后，将其学习对齐至潜在空间。此外，我们引入3D面部先验，使模型能够以灵活且3D一致的方式控制人体头部。大量定性与定量分析表明，CapHuman可生成身份保持良好、照片级真实且高保真的人像，具备丰富的语义表征和多样的头部表现形式，性能优于现有基线模型。代码与模型权重将在https://github.com/VamosC/CapHuman 开源。