In this paper, we propose a 3D asset-referenced diffusion model for image generation, exploring how to integrate 3D assets into image diffusion models. Existing reference-based image generation methods leverage large-scale pretrained diffusion models and demonstrate strong capability in generating diverse images conditioned on a single reference image. However, these methods are limited to single-image references and cannot leverage 3D assets, constraining their practical versatility. To address this gap, we present a cross-domain diffusion model with dual-branch perception that leverages multi-view RGB images and point maps of 3D assets to jointly model their colors and canonical-space coordinates, achieving precise consistency between generated images and the 3D references. Our spatially aligned dual-branch generation architecture and domain-decoupled generation mechanism ensure the simultaneous generation of two spatially aligned but content-disentangled outputs, RGB images and point maps, linking 2D image attributes with 3D asset attributes. Experiments show that our approach effectively uses 3D assets as references to produce images consistent with the given assets, opening new possibilities for combining diffusion models with 3D content creation.

翻译：本文提出一种用于图像生成的3D资产参考扩散模型，旨在探索如何将3D资产整合到图像扩散模型中。现有的基于参考的图像生成方法利用大规模预训练扩散模型，展现了在单张参考图像条件下生成多样化图像的强大能力。然而，这些方法仅限于单图像参考，无法利用3D资产，限制了其实际应用的灵活性。为填补这一空白，我们提出一种具有双分支感知的跨域扩散模型，该模型利用3D资产的多视角RGB图像与点云图，联合建模其颜色与规范空间坐标，从而实现生成图像与3D参考之间的精确一致性。我们的空间对齐双分支生成架构与解耦域生成机制，确保了RGB图像和点云图这两种空间对齐但内容解耦的输出能够同时生成，从而将2D图像属性与3D资产属性相联结。实验表明，我们的方法能有效利用3D资产作为参考，生成与给定资产一致的图像，为扩散模型与3D内容创作的结合开辟了新的可能性。