2D diffusion model, which often contains unwanted baked-in shading effects and results in unrealistic rendering effects in the downstream applications. Generating Physically Based Rendering (PBR) materials instead of just RGB textures would be a promising solution. However, directly distilling the PBR material parameters from 2D diffusion models still suffers from incorrect material decomposition, such as baked-in shading effects in albedo. We introduce DreamMat, an innovative approach to resolve the aforementioned problem, to generate high-quality PBR materials from text descriptions. We find out that the main reason for the incorrect material distillation is that large-scale 2D diffusion models are only trained to generate final shading colors, resulting in insufficient constraints on material decomposition during distillation. To tackle this problem, we first finetune a new light-aware 2D diffusion model to condition on a given lighting environment and generate the shading results on this specific lighting condition. Then, by applying the same environment lights in the material distillation, DreamMat can generate high-quality PBR materials that are not only consistent with the given geometry but also free from any baked-in shading effects in albedo. Extensive experiments demonstrate that the materials produced through our methods exhibit greater visual appeal to users and achieve significantly superior rendering quality compared to baseline methods, which are preferable for downstream tasks such as game and film production.

翻译：现有的文本到材质生成方法通常依赖于2D扩散模型，这类模型往往包含不期望的预烘焙着色效果，导致在下游应用中产生不真实的渲染效果。生成基于物理的渲染（PBR）材质而非仅生成RGB纹理将是一种有前景的解决方案。然而，直接从2D扩散模型中蒸馏提取PBR材质参数仍存在材质分解错误的问题，例如反照率中残留的预烘焙着色效应。本文提出DreamMat这一创新方法以解决上述问题，实现从文本描述生成高质量的PBR材质。我们发现材质蒸馏错误的主要原因是：大规模2D扩散模型仅训练生成最终着色颜色，导致蒸馏过程中对材质分解的约束不足。为解决该问题，我们首先微调了一个新型光照感知2D扩散模型，使其能够根据给定光照环境生成特定光照条件下的着色结果。随后，通过在材质蒸馏过程中应用相同的环境光照，DreamMat能够生成与给定几何结构一致、且反照率中完全不含预烘焙着色效应的高质量PBR材质。大量实验表明，相较于基线方法，通过我们方法生成的材质在视觉上更受用户青睐，并实现了显著优越的渲染质量，尤其适用于游戏与电影制作等下游任务。