Reconstructing textured meshes from colored point clouds is an important but challenging task. Most existing methods yield blurry-looking textures or rely on 3D training data that are hard to acquire. Regarding this, we propose PointDreamer, a novel framework for textured mesh reconstruction from colored point cloud via diffusion-based 2D inpainting. Specifically, we first reconstruct an untextured mesh. Next, we project the input point cloud into 2D space to generate sparse multi-view images, and then inpaint empty pixels utilizing a pre-trained 2D diffusion model. After that, we unproject the colors of the inpainted dense images onto the untextured mesh, thus obtaining the final textured mesh. This project-inpaint-unproject pipeline bridges the gap between 3D point clouds and 2D diffusion models for the first time. Thanks to the powerful 2D diffusion model pre-trained on extensive 2D data, PointDreamer reconstructs clear, high-quality textures with high robustness to sparse or noisy input. Also, it's zero-shot requiring no extra training. In addition, we design Non-Border-First unprojection strategy to address the border-area inconsistency issue, which is less explored but commonly-occurred in methods that generate 3D textures from multiview images. Extensive qualitative and quantitative experiments on various synthetic and real-scanned datasets show the SoTA performance of PointDreamer, by significantly outperforming baseline methods with 30% improvement in LPIPS score (from 0.118 to 0.068). Code at: https://github.com/YuQiao0303/PointDreamer.

翻译：从彩色点云重建纹理网格是一项重要但具有挑战性的任务。现有方法大多产生模糊的纹理，或依赖于难以获取的三维训练数据。为此，我们提出PointDreamer，一种通过基于扩散的二维修复从彩色点云重建纹理网格的新框架。具体而言，我们首先重建一个无纹理网格。接着，将输入点云投影至二维空间以生成稀疏多视角图像，并利用预训练的二维扩散模型修复空白像素。随后，将修复后稠密图像的颜色反投影至无纹理网格，从而获得最终的纹理网格。该“投影-修复-反投影”流程首次在三维点云与二维扩散模型之间建立了桥梁。得益于在大量二维数据上预训练的强⼤二维扩散模型，PointDreamer能够重建清晰、⾼质量的纹理，并对稀疏或含噪声输入具有高鲁棒性。同时，该方法为零样本⽅法，无需额外训练。此外，我们设计了“非边界优先”反投影策略，以解决边界区域不一致问题——该问题在基于多视图图像生成三维纹理的方法中普遍存在但鲜有研究。在多种合成与真实扫描数据集上的大量定性与定量实验表明，PointDreamer取得了最先进的性能，其LPIPS分数较基线方法显著提升30%（从0.118降至0.068）。代码位于：https://github.com/YuQiao0303/PointDreamer。