Large pretrained diffusion models can provide strong priors beneficial for many graphics applications. However, generative applications such as neural rendering and inverse methods such as SVBRDF estimation and intrinsic image decomposition require additional input or output channels. Current solutions for channel expansion are often application specific and these solutions can be difficult to adapt to different diffusion models or new tasks. This paper introduces Teamwork: a flexible and efficient unified solution for jointly increasing the number of input and output channels as well as adapting a pretrained diffusion model to new tasks. Teamwork achieves channel expansion without altering the pretrained diffusion model architecture by coordinating and adapting multiple instances of the base diffusion model (\ie, teammates). We employ a novel variation of Low Rank-Adaptation (LoRA) to jointly address both adaptation and coordination between the different teammates. Furthermore Teamwork supports dynamic (de)activation of teammates. We demonstrate the flexibility and efficiency of Teamwork on a variety of generative and inverse graphics tasks such as inpainting, single image SVBRDF estimation, intrinsic decomposition, neural shading, and intrinsic image synthesis.

翻译：大型预训练扩散模型能够为众多图形学应用提供强有力的先验。然而，神经渲染等生成式应用以及SVBRDF估计、本征图像分解等逆向方法通常需要额外的输入或输出通道。现有的通道扩展方案往往针对特定应用设计，且难以适配不同的扩散模型或新任务。本文提出Teamwork：一种灵活高效的一体化解决方案，能够联合扩展输入输出通道数量，并将预训练扩散模型适配至新任务。Teamwork通过协调并适配多个基础扩散模型实例（即“协作者”）实现通道扩展，而无需改变预训练扩散模型的原始架构。我们采用一种新颖的低秩适配（LoRA）变体方法，协同处理不同协作者之间的适配与协调问题。此外，Teamwork支持协作者的动态（去）激活功能。我们通过修复、单图像SVBRDF估计、本征分解、神经着色及本征图像合成等多种生成式与逆向图形学任务，验证了Teamwork方案的灵活性与高效性。