We present SCULPT, a novel 3D generative model for clothed and textured 3D meshes of humans. Specifically, we devise a deep neural network that learns to represent the geometry and appearance distribution of clothed human bodies. Training such a model is challenging, as datasets of textured 3D meshes for humans are limited in size and accessibility. Our key observation is that there exist medium-sized 3D scan datasets like CAPE, as well as large-scale 2D image datasets of clothed humans and multiple appearances can be mapped to a single geometry. To effectively learn from the two data modalities, we propose an unpaired learning procedure for pose-dependent clothed and textured human meshes. Specifically, we learn a pose-dependent geometry space from 3D scan data. We represent this as per vertex displacements w.r.t. the SMPL model. Next, we train a geometry conditioned texture generator in an unsupervised way using the 2D image data. We use intermediate activations of the learned geometry model to condition our texture generator. To alleviate entanglement between pose and clothing type, and pose and clothing appearance, we condition both the texture and geometry generators with attribute labels such as clothing types for the geometry, and clothing colors for the texture generator. We automatically generated these conditioning labels for the 2D images based on the visual question answering model BLIP and CLIP. We validate our method on the SCULPT dataset, and compare to state-of-the-art 3D generative models for clothed human bodies. We will release the codebase for research purposes.

翻译：我们提出SCULPT，一种用于穿衣且具备纹理的人体三维网格的新型生成模型。具体而言，我们设计了一种深度神经网络，学习表示穿衣人体的几何与外观分布。训练此类模型具有挑战性，因为带纹理的三维人体网格数据集在规模和可获取性上均受限。我们的关键观察在于：存在中等规模的三维扫描数据集（如CAPE），以及大规模穿衣人体的二维图像数据集，且多种外观可映射至单一几何结构。为有效从两种数据模态学习，我们提出一种针对姿态相关的穿衣与纹理化人体网格的非配对学习流程。具体地，我们从三维扫描数据中学习姿态依赖的几何空间，并将其表示为相对于SMPL模型的逐顶点位移。随后，我们以无监督方式利用二维图像数据训练几何条件纹理生成器，使用所学几何模型的中间层激活特征作为纹理生成器的条件。为缓解姿态与服装类型、姿态与服装外观之间的纠缠，我们对纹理与几何生成器均施加属性标签条件：几何生成器以服装类型为条件，纹理生成器以服装颜色为条件。我们基于视觉问答模型BLIP和CLIP为二维图像自动生成这些条件标签。我们在SCULPT数据集上验证了方法，并与最先进的穿衣人体三维生成模型进行对比。我们将开源代码库以供研究用途。