Learning to generate textures for a novel 3D mesh given a collection of 3D meshes and real-world 2D images is an important problem with applications in various domains such as 3D simulation, augmented and virtual reality, gaming, architecture, and design. Existing solutions either do not produce high-quality textures or deform the original high-resolution input mesh topology into a regular grid to make this generation easier but also lose the original mesh topology. In this paper, we present a novel framework called the 3DTextureTransformer that enables us to generate high-quality textures without deforming the original, high-resolution input mesh. Our solution, a hybrid of geometric deep learning and StyleGAN-like architecture, is flexible enough to work on arbitrary mesh topologies and also easily extensible to texture generation for point cloud representations. Our solution employs a message-passing framework in 3D in conjunction with a StyleGAN-like architecture for 3D texture generation. The architecture achieves state-of-the-art performance among a class of solutions that can learn from a collection of 3D geometry and real-world 2D images while working with any arbitrary mesh topology.

翻译：针对给定三维网格集合与实际二维图像来学习生成新三维网格的纹理，是一个涉及三维仿真、增强/虚拟现实、游戏、建筑及设计等多个领域的重要问题。现有方案要么无法生成高质量纹理，要么为简化生成过程而将原始高分辨率输入网格拓扑变形为规则网格，从而丢失原始网格拓扑。本文提出一种名为3DTextureTransformer的新型框架，能够在不改变原始高分辨率输入网格的前提下生成高质量纹理。该方案融合了几何深度学习与类StyleGAN架构，可灵活适用于任意网格拓扑，并能轻松扩展至点云表示的纹理生成。我们采用三维消息传递框架与类StyleGAN架构相结合的方式实现三维纹理生成。在能够从三维几何体集合及真实二维图像中学习的同类方案中，该架构在处理任意网格拓扑时达到了最先进性能。