Material reconstruction from a photograph is a key component of 3D content creation democratization. We propose to formulate this ill-posed problem as a controlled synthesis one, leveraging the recent progress in generative deep networks. We present ControlMat, a method which, given a single photograph with uncontrolled illumination as input, conditions a diffusion model to generate plausible, tileable, high-resolution physically-based digital materials. We carefully analyze the behavior of diffusion models for multi-channel outputs, adapt the sampling process to fuse multi-scale information and introduce rolled diffusion to enable both tileability and patched diffusion for high-resolution outputs. Our generative approach further permits exploration of a variety of materials which could correspond to the input image, mitigating the unknown lighting conditions. We show that our approach outperforms recent inference and latent-space-optimization methods, and carefully validate our diffusion process design choices. Supplemental materials and additional details are available at: https://gvecchio.com/controlmat/.

翻译：从单张照片重建材质是三维内容创作民主化的关键环节。我们提出将这一不适定问题转化为受控合成问题，借助生成式深度网络的最新进展。我们提出ControlMat方法：该方法以单张光照条件未知的照片为输入，引导扩散模型生成合理、可平铺、高分辨率的物理基数字材质。我们深入分析了扩散模型在多通道输出中的行为特性，调整采样过程以融合多尺度信息，并引入滚转扩散以实现高分辨率输出的可平铺性与分块扩散。这种生成式方法还能探索与输入图像对应的多种材质可能性，从而缓解未知光照条件的影响。实验表明，我们的方法优于近期提出的推理与隐空间优化方法，并验证了扩散模型设计选择的合理性。补充材料与更多细节详见：https://gvecchio.com/controlmat/。