Current 3D content generation approaches build on diffusion models that output RGB images. Modern graphics pipelines, however, require physically-based rendering (PBR) material properties. We propose to model the PBR image distribution directly, avoiding photometric inaccuracies in RGB generation and the inherent ambiguity in extracting PBR from RGB. Existing paradigms for cross-modal fine-tuning are not suited for PBR generation due to both a lack of data and the high dimensionality of the output modalities: we overcome both challenges by retaining a frozen RGB model and tightly linking a newly trained PBR model using a novel cross-network communication paradigm. As the base RGB model is fully frozen, the proposed method does not risk catastrophic forgetting during fine-tuning and remains compatible with techniques such as IPAdapter pretrained for the base RGB model. We validate our design choices, robustness to data sparsity, and compare against existing paradigms with an extensive experimental section.

翻译：当前的3D内容生成方法依赖于输出RGB图像的扩散模型。然而，现代图形管线需要基于物理渲染（PBR）材质属性。我们提出直接建模PBR图像分布，以避免RGB生成中的光度不准确性以及从RGB中提取PBR时固有的歧义性。现有的跨模态微调范式由于缺乏数据以及输出模态的高维度性，不适用于PBR生成：我们通过保留冻结的RGB模型，并利用一种新颖的跨网络通信范式，紧密连接新训练的PBR模型，从而克服了这两个挑战。由于基础RGB模型完全冻结，所提方法在微调过程中不会出现灾难性遗忘的风险，并且与预训练用于基础RGB模型的IPAdapter等技术保持兼容。我们通过广泛的实验部分验证了我们的设计选择、对数据稀疏性的鲁棒性，并对比了现有范式的性能。