We propose X-Portrait, an innovative conditional diffusion model tailored for generating expressive and temporally coherent portrait animation. Specifically, given a single portrait as appearance reference, we aim to animate it with motion derived from a driving video, capturing both highly dynamic and subtle facial expressions along with wide-range head movements. As its core, we leverage the generative prior of a pre-trained diffusion model as the rendering backbone, while achieve fine-grained head pose and expression control with novel controlling signals within the framework of ControlNet. In contrast to conventional coarse explicit controls such as facial landmarks, our motion control module is learned to interpret the dynamics directly from the original driving RGB inputs. The motion accuracy is further enhanced with a patch-based local control module that effectively enhance the motion attention to small-scale nuances like eyeball positions. Notably, to mitigate the identity leakage from the driving signals, we train our motion control modules with scaling-augmented cross-identity images, ensuring maximized disentanglement from the appearance reference modules. Experimental results demonstrate the universal effectiveness of X-Portrait across a diverse range of facial portraits and expressive driving sequences, and showcase its proficiency in generating captivating portrait animations with consistently maintained identity characteristics.

翻译：我们提出X-Portrait，一种创新的条件扩散模型，专为生成富有表现力且时序连贯的肖像动画而设计。具体而言，给定单张肖像作为外观参考，我们旨在利用驱动视频提取的运动信息对其进行动画化，同时捕捉高度动态和微妙的面部表情以及大范围头部运动。其核心在于，我们利用预训练扩散模型的生成先验作为渲染主干，同时通过ControlNet框架内新颖的控制信号实现细粒度的头部姿态与表情控制。相较于传统基于面部关键点等粗粒度显式控制方法，我们的运动控制模块通过学习直接从原始驱动RGB输入中解析动态信息。通过引入基于图像块的局部控制模块，运动精度得到进一步提升，该模块能有效增强对眼球位置等小尺度细微特征的运动注意力。值得注意的是，为减少驱动信号中的身份信息泄露，我们使用尺度增强的跨身份图像训练运动控制模块，确保其与外观参考模块实现最大程度的解耦。实验结果表明，X-Portrait在不同类型面部肖像和富有表现力的驱动序列上均具有普适有效性，并展现出在保持身份特征一致性的同时生成引人入胜的肖像动画的卓越能力。