We present Paint-it, a text-driven high-fidelity texture map synthesis method for 3D meshes via neural re-parameterized texture optimization. Paint-it synthesizes texture maps from a text description by synthesis-through-optimization, exploiting the Score-Distillation Sampling (SDS). We observe that directly applying SDS yields undesirable texture quality due to its noisy gradients. We reveal the importance of texture parameterization when using SDS. Specifically, we propose Deep Convolutional Physically-Based Rendering (DC-PBR) parameterization, which re-parameterizes the physically-based rendering (PBR) texture maps with randomly initialized convolution-based neural kernels, instead of a standard pixel-based parameterization. We show that DC-PBR inherently schedules the optimization curriculum according to texture frequency and naturally filters out the noisy signals from SDS. In experiments, Paint-it obtains remarkable quality PBR texture maps within 15 min., given only a text description. We demonstrate the generalizability and practicality of Paint-it by synthesizing high-quality texture maps for large-scale mesh datasets and showing test-time applications such as relighting and material control using a popular graphics engine. Project page: https://kim-youwang.github.io/paint-it

翻译：我们提出Paint-it，一种通过神经重参数化纹理优化实现文本驱动的三维网格高保真纹理图合成方法。Paint-it利用得分蒸馏采样（SDS）技术，通过合成即优化策略从文本描述生成纹理图。我们观察到直接应用SDS会因其噪声梯度导致纹理质量不佳，并揭示了在使用SDS时纹理参数化的重要性。具体而言，我们提出深度卷积物理渲染（DC-PBR）参数化方法，该方法采用随机初始化的卷积神经核重参数化物理渲染（PBR）纹理图，而非标准的像素级参数化。研究表明，DC-PBR能够根据纹理频率内在调整优化进程，并自然滤除SDS中的噪声信号。实验表明，仅需文本描述，Paint-it便可在15分钟内生成高质量的PBR纹理图。我们通过为大规模网格数据集合成高质量纹理图，并展示基于流行图形引擎的实时重光照与材质控制等测试时应用，验证了Paint-it的泛化性与实用性。项目主页：https://kim-youwang.github.io/paint-it