We introduce SyncLipMAE, a self-supervised pretraining framework for talking-face video that learns synchronization-aware and transferable facial dynamics from unlabeled audio-visual streams. Our approach couples masked visual modeling with cross-modal contrastive alignment and employs three per-frame prompt tokens that explicitly encode the essential factors of a talking-face frame - identity, vocal motion (speech-synchronized facial dynamics), and ambient motion (audio-agnostic movements such as blinks and head pose). The contrastive objective uses time-aligned vocal-motion and audio tokens as positives and misaligned pairs as negatives, driving both modalities into a shared embedding space and yielding token-level audio-visual stream synchronization. After pretraining, the aligned audio tokens together with the visual prompt tokens (identity, vocal motion, ambient motion) form a unified interface for four disparate downstream settings: (i) audio-visual stream synchronization; (ii) facial emotion and head/face action recognition; (iii) visual speech recognition; and (iv) visual dubbing, for which we enable indistinguishable audio- or video-driven control within a single model. Across four task families that require distinct capabilities, SyncLipMAE achieves state-of-the-art results, underscoring the effectiveness of synchronization-aware, factorized self-supervised pretraining.

翻译：我们提出了SyncLipMAE，一种用于说话人脸视频的自监督预训练框架，该框架从无标注的视听流中学习具有同步感知能力且可迁移的面部动态。我们的方法将掩码视觉建模与跨模态对比对齐相结合，并采用三个逐帧提示令牌，它们显式地编码了说话人脸帧的三个基本要素——身份、发声动作（与语音同步的面部动态）以及环境动作（与音频无关的动作，如眨眼和头部姿态）。对比学习目标以时间对齐的发声动作令牌和音频令牌作为正样本，以未对齐的配对作为负样本，从而将两种模态驱动到一个共享的嵌入空间中，并实现令牌级别的视听流同步。预训练完成后，对齐的音频令牌与视觉提示令牌（身份、发声动作、环境动作）共同构成了一个统一的接口，适用于四种不同的下游任务场景：（i）视听流同步；（ii）面部表情及头部/面部动作识别；（iii）视觉语音识别；以及（iv）视觉配音，我们使得单一模型能够实现难以区分的音频驱动或视频驱动控制。在需要不同能力的四个任务族上，SyncLipMAE均取得了最先进的结果，这凸显了具有同步感知能力的、解耦式自监督预训练的有效性。