We propose ClipFace, a novel self-supervised approach for text-guided editing of textured 3D morphable model of faces. Specifically, we employ user-friendly language prompts to enable control of the expressions as well as appearance of 3D faces. We leverage the geometric expressiveness of 3D morphable models, which inherently possess limited controllability and texture expressivity, and develop a self-supervised generative model to jointly synthesize expressive, textured, and articulated faces in 3D. We enable high-quality texture generation for 3D faces by adversarial self-supervised training, guided by differentiable rendering against collections of real RGB images. Controllable editing and manipulation are given by language prompts to adapt texture and expression of the 3D morphable model. To this end, we propose a neural network that predicts both texture and expression latent codes of the morphable model. Our model is trained in a self-supervised fashion by exploiting differentiable rendering and losses based on a pre-trained CLIP model. Once trained, our model jointly predicts face textures in UV-space, along with expression parameters to capture both geometry and texture changes in facial expressions in a single forward pass. We further show the applicability of our method to generate temporally changing textures for a given animation sequence.

翻译：我们提出ClipFace，一种新颖的自监督方法，用于文本引导的可纹理化三维人脸可变形模型编辑。具体而言，我们采用用户友好的语言提示，以实现对三维人脸表情及外观的控制。我们利用三维可变形模型在几何表达力上的优势（尽管其本身在可控性和纹理表现力方面存在局限），并开发了一种自监督生成模型，以联合合成具有表现力、带纹理且可关节运动的三维人脸。通过对抗式自监督训练，并结合基于真实RGB图像集合的可微渲染指导，我们实现了三维人脸的高质量纹理生成。语言提示赋予模型可控的编辑与操作能力，可调整三维可变形模型的纹理与表情。为此，我们提出一种神经网络，用于预测可变形模型的纹理和表情潜在编码。该模型通过利用可微渲染和基于预训练CLIP模型的损失函数进行自监督训练。训练完成后，我们的模型可在单次前向传播中联合预测UV空间中的面部纹理及表情参数，从而捕捉面部表情中的几何与纹理变化。我们进一步展示了该方法在生成给定动画序列随时间变化的纹理方面的适用性。