Neural radiance fields (NeRF) based methods have shown amazing performance in synthesizing 3D-consistent photographic images, but fail to generate multi-view portrait drawings. The key is that the basic assumption of these methods -- a surface point is consistent when rendered from different views -- doesn't hold for drawings. In a portrait drawing, the appearance of a facial point may changes when viewed from different angles. Besides, portrait drawings usually present little 3D information and suffer from insufficient training data. To combat this challenge, in this paper, we propose a Semantic-Aware GEnerator (SAGE) for synthesizing multi-view portrait drawings. Our motivation is that facial semantic labels are view-consistent and correlate with drawing techniques. We therefore propose to collaboratively synthesize multi-view semantic maps and the corresponding portrait drawings. To facilitate training, we design a semantic-aware domain translator, which generates portrait drawings based on features of photographic faces. In addition, use data augmentation via synthesis to mitigate collapsed results. We apply SAGE to synthesize multi-view portrait drawings in diverse artistic styles. Experimental results show that SAGE achieves significantly superior or highly competitive performance, compared to existing 3D-aware image synthesis methods. The codes are available at https://github.com/AiArt-HDU/SAGE.
翻译:基于神经辐射场(NeRF)的方法在合成3D一致的摄影图像方面展现出惊人的性能,但无法生成多视角人像素描。关键在于这些方法的基本假设——表面点从不同视角渲染时是一致的——不适用于素描。在人像素描中,面部点的外观可能因观察角度不同而改变。此外,人像素描通常缺乏3D信息,且面临训练数据不足的问题。为解决这一挑战,本文提出了一种语义感知生成器(SAGE),用于合成多视角人像素描。我们的动机在于面部语义标签具有视角一致性,并与素描技法相关。因此,我们提出协同合成多视角语义图及其对应的人像素描。为促进训练,我们设计了一个语义感知域转换器,基于摄影人脸特征生成人像素描。此外,通过合成数据增强来规避结果坍缩问题。我们将SAGE应用于合成多种艺术风格的多视角人像素描。实验结果表明,与现有3D感知图像合成方法相比,SAGE实现了显著优越或极具竞争力的性能。代码已开源至https://github.com/AiArt-HDU/SAGE。