Current 3D content generation builds on generative models that output RGB images. Modern graphics pipelines, however, require physically-based rendering (PBR) material properties. We propose to model the PBR image distribution directly to avoid photometric inaccuracies in RGB generation and the inherent ambiguity in extracting PBR from RGB. Existing paradigms for cross-modal finetuning are not suited for PBR generation due to a lack of data and the high dimensionality of the output modalities: we overcome both challenges by retaining a frozen RGB model and tightly linking a newly trained PBR model using a novel cross-network communication paradigm. As the base RGB model is fully frozen, the proposed method does not risk catastrophic forgetting during finetuning and remains compatible with techniques such as IPAdapter pretrained for the base RGB model. We validate our design choices, robustness to data sparsity, and compare against existing paradigms with an extensive experimental section.

翻译：当前三维内容生成技术依赖于输出RGB图像的生成模型。然而，现代图形管线需要基于物理的渲染（PBR）材质属性。我们提出直接对PBR图像分布进行建模，以避免RGB生成中的光度误差以及从RGB提取PBR时固有的歧义性。由于数据匮乏和输出模态的高维特性，现有的跨模态微调范式并不适用于PBR生成：我们通过保留冻结的RGB模型，并利用一种新颖的跨网络通信机制紧密连接新训练的PBR模型，从而克服了这两项挑战。由于基础RGB模型完全保持冻结，所提方法在微调过程中不会出现灾难性遗忘，并且仍能兼容为基础RGB模型预训练的技术（如IPAdapter）。通过大量实验，我们验证了设计选择的合理性、对数据稀疏性的鲁棒性，并与现有范式进行了对比。