Coarse architectural models are often generated at scales ranging from individual buildings to scenes for downstream applications such as Digital Twin City, Metaverse, LODs, etc. Such piece-wise planar models can be abstracted as twins from 3D dense reconstructions. However, these models typically lack realistic texture relative to the real building or scene, making them unsuitable for vivid display or direct reference. In this paper, we present TwinTex, the first automatic texture mapping framework to generate a photo-realistic texture for a piece-wise planar proxy. Our method addresses most challenges occurring in such twin texture generation. Specifically, for each primitive plane, we first select a small set of photos with greedy heuristics considering photometric quality, perspective quality and facade texture completeness. Then, different levels of line features (LoLs) are extracted from the set of selected photos to generate guidance for later steps. With LoLs, we employ optimization algorithms to align texture with geometry from local to global. Finally, we fine-tune a diffusion model with a multi-mask initialization component and a new dataset to inpaint the missing region. Experimental results on many buildings, indoor scenes and man-made objects of varying complexity demonstrate the generalization ability of our algorithm. Our approach surpasses state-of-the-art texture mapping methods in terms of high-fidelity quality and reaches a human-expert production level with much less effort. Project page: https://vcc.tech/research/2023/TwinTex.

翻译：粗粒度建筑模型通常以从单体建筑到场景的尺度生成，服务于数字孪生城市、元宇宙、LOD（细节层次模型）等下游应用。此类分段平面模型可从三维稠密重建中抽象为"孪生体"，然而相较于真实建筑或场景，这些模型往往缺乏逼真纹理，难以实现生动展示或直接参考。本文提出TwinTex——首个面向分段平面代理的自动纹理映射框架，旨在生成照片级真实纹理。本方法解决了孪生纹理生成中的主要挑战：具体而言，对每个基本平面，我们首先采用考虑光度质量、透视质量与立面纹理完整性的贪心启发式策略，从候选照片中精选少量图像；随后从选定照片集中提取多层级线特征（LoLs），为后续处理提供引导；基于LoLs，我们运用优化算法实现从局部到全局的纹理与几何对齐；最后，通过包含多掩码初始化组件的扩散模型微调及新数据集构建，对缺失区域进行修复。在涵盖不同复杂程度的建筑、室内场景与人造物体上的实验结果表明，本算法具备泛化能力，在保真度上超越现有最优纹理映射方法，以更低工作量达到专业级水准。项目主页：https://vcc.tech/research/2023/TwinTex