Digital humans are fundamental to immersive interaction, yet creating a unified model for holistic modalities, including text, audio, motion, and visual content, remains an open challenge. In this paper, we present Archon, a fully pretrained, human-centric unified multimodal model for holistic avatar generation. Archon unifies seven modalities with modality-specific tokenizers, and a native autoregressive unified multimodal model pretrained on synchronized modalities and 72 diverse tasks to model holistic joint distributions. To address the token explosion challenge in high-fidelity talking videos, we introduce a memory-efficient semantic video reparameterization, achieving 4x token reduction while preserving fine-grained dynamics, coupled with a semantic-driven video diffusion decoder. We further propose a "Thinking in Modality" that decomposes ambiguous cross-modal tasks into stepwise thinking in an alternative chain of modality, progressively enhancing fidelity and controllability. Extensive experiments demonstrate that Archon achieves superior or comparable performance across diverse digital human generation tasks, validating the effectiveness of our unified framework. Project page: https://zju3dv.github.io/archon/.
翻译:摘要:数字人类是实现沉浸式交互的基础,然而构建一个涵盖文本、语音、动作与视觉内容等全维度模态的统一模型仍是未解决的关键挑战。本文提出Archon——一个完全预训练的、以人为中心的统一多模态全维化身生成模型。该模型通过模态专用分词器统一七种模态,并利用原生自回归统一多模态模型在同步模态与72种多样化任务上进行预训练,以建模全维联合分布。针对高保真对话视频中的Token爆炸问题,我们提出一种内存高效的语义视频重参数化方法,在保留细粒度动力学特征的同时实现4倍Token压缩,并配合语义驱动的视频扩散解码器。进一步提出"模态思维"机制,将歧义跨模态任务分解为交替模态链中的逐步推理过程,逐步提升保真度与可控性。大量实验表明,Archon在多种数字人生成任务中展现出优越或相当的性能,验证了统一框架的有效性。项目页面:https://zju3dv.github.io/archon/。