Generating face image with specific gaze information has attracted considerable attention. Existing approaches typically input gaze values directly for face generation, which is unnatural and requires annotated gaze datasets for training, thereby limiting its application. In this paper, we present a novel gaze-controllable face generation task. Our approach inputs textual descriptions that describe human gaze and head behavior and generates corresponding face images. Our work first introduces a text-of-gaze dataset containing over 90k text descriptions spanning a dense distribution of gaze and head poses. We further propose a gaze-controllable text-to-face method. Our method contains a sketch-conditioned face diffusion module and a model-based sketch diffusion module. We define a face sketch based on facial landmarks and eye segmentation map. The face diffusion module generates face images from the face sketch, and the sketch diffusion module employs a 3D face model to generate face sketch from text description. Experiments on the FFHQ dataset show the effectiveness of our method. We will release our dataset and code for future research.

翻译：生成具有特定注视信息的人脸图像受到广泛关注。现有方法通常直接输入注视值进行人脸生成，这种方式不够自然且需要标注的注视数据集进行训练，从而限制了其应用。本文提出了一种新颖的注视可控人脸生成任务，通过输入描述人类注视和头部行为的文本描述，生成对应的人脸图像。我们的工作首先引入了一个包含超过9万条文本描述的注视文本数据集，这些描述覆盖了密集分布的注视方向和头部姿态。我们进一步提出了一种基于文本的注视可控人脸生成方法，该方法包含基于草图条件的人脸扩散模块和基于模型的草图扩散模块。我们根据面部关键点和眼睛分割图定义人脸草图。人脸扩散模块从人脸草图生成人脸图像，而草图扩散模块则利用3D人脸模型从文本描述生成人脸草图。在FFHQ数据集上的实验证明了我们方法的有效性。我们将发布数据集和代码以供后续研究使用。