In this paper, we propose a method to extract physically-based rendering (PBR) materials from a single real-world image. We do so in two steps: first, we map regions of the image to material concepts using a diffusion model, which allows the sampling of texture images resembling each material in the scene. Second, we benefit from a separate network to decompose the generated textures into Spatially Varying BRDFs (SVBRDFs), providing us with materials ready to be used in rendering applications. Our approach builds on existing synthetic material libraries with SVBRDF ground truth, but also exploits a diffusion-generated RGB texture dataset to allow generalization to new samples using unsupervised domain adaptation (UDA). Our contributions are thoroughly evaluated on synthetic and real-world datasets. We further demonstrate the applicability of our method for editing 3D scenes with materials estimated from real photographs. The code and models will be made open-source. Project page: https://astra-vision.github.io/MaterialPalette/

翻译：本文提出了一种从单张真实世界图像中提取基于物理渲染（PBR）材质的方法。该方法分为两步：首先，利用扩散模型将图像区域映射至材质概念，从而能够对场景中各材质对应的纹理图像进行采样；其次，借助独立的神经网络将生成的纹理分解为空间变化的双向反射分布函数（SVBRDF），进而获得可直接用于渲染应用的材质。本方法基于现有包含SVBRDF真实标签的合成材质库，同时利用扩散生成的RGB纹理数据集，通过无监督域适应（UDA）实现对新型样本的泛化。我们在合成数据集与真实数据集上对方法进行了全面评估，并进一步展示了该方法在基于真实照片估计材质以编辑三维场景中的应用。代码与模型将开源。项目页面：https://astra-vision.github.io/MaterialPalette/