Large-scale pre-trained vision-language models allow for the zero-shot text-based generation of 3D avatars. The previous state-of-the-art method utilized CLIP to supervise neural implicit models that reconstructed a human body mesh. However, this approach has two limitations. Firstly, the lack of avatar-specific models can cause facial distortion and unrealistic clothing in the generated avatars. Secondly, CLIP only provides optimization direction for the overall appearance, resulting in less impressive results. To address these limitations, we propose AvatarFusion, the first framework to use a latent diffusion model to provide pixel-level guidance for generating human-realistic avatars while simultaneously segmenting clothing from the avatar's body. AvatarFusion includes the first clothing-decoupled neural implicit avatar model that employs a novel Dual Volume Rendering strategy to render the decoupled skin and clothing sub-models in one space. We also introduce a novel optimization method, called Pixel-Semantics Difference-Sampling (PS-DS), which semantically separates the generation of body and clothes, and generates a variety of clothing styles. Moreover, we establish the first benchmark for zero-shot text-to-avatar generation. Our experimental results demonstrate that our framework outperforms previous approaches, with significant improvements observed in all metrics. Additionally, since our model is clothing-decoupled, we can exchange the clothes of avatars. Code will be available on Github.

翻译：大规模预训练视觉-语言模型支持基于文本的零样本三维化身生成。先前最先进的方法利用CLIP监督神经隐式模型，重建人体网格。然而，该方法存在两个局限性：首先，缺乏化身专用模型会导致生成的面部扭曲和衣物不真实；其次，CLIP仅提供整体外观的优化方向，导致效果欠佳。为解决这些问题，我们提出AvatarFusion——首个利用潜在扩散模型为生成逼真化身提供像素级引导，同时实现衣物与化身身体分割的框架。AvatarFusion包含首个衣物解耦神经隐式化身模型，采用新颖的双体渲染策略在同一空间中渲染解耦的皮肤和衣物子模型。我们提出名为像素语义差异采样（PS-DS）的新型优化方法，该方法从语义上分离身体与衣物的生成过程，并生成多种衣物风格。此外，我们建立了首个零样本文本到化身生成的基准。实验结果表明，我们的框架在所有指标上均显著超越先前方法。由于模型实现了衣物解耦，我们还可以交换化身的衣物。代码将在GitHub上提供。