As pretrained text-to-image diffusion models become increasingly powerful, recent efforts have been made to distill knowledge from these text-to-image pretrained models for optimizing a text-guided 3D model. Most of the existing methods generate a holistic 3D model from a plain text input. This can be problematic when the text describes a complex scene with multiple objects, because the vectorized text embeddings are inherently unable to capture a complex description with multiple entities and relationships. Holistic 3D modeling of the entire scene further prevents accurate grounding of text entities and concepts. To address this limitation, we propose GraphDreamer, a novel framework to generate compositional 3D scenes from scene graphs, where objects are represented as nodes and their interactions as edges. By exploiting node and edge information in scene graphs, our method makes better use of the pretrained text-to-image diffusion model and is able to fully disentangle different objects without image-level supervision. To facilitate modeling of object-wise relationships, we use signed distance fields as representation and impose a constraint to avoid inter-penetration of objects. To avoid manual scene graph creation, we design a text prompt for ChatGPT to generate scene graphs based on text inputs. We conduct both qualitative and quantitative experiments to validate the effectiveness of GraphDreamer in generating high-fidelity compositional 3D scenes with disentangled object entities.

翻译：随着预训练文本到图像扩散模型的日益强大，近年来研究者致力于从这些文本到图像预训练模型中提炼知识，以优化文本引导的三维模型。现有方法大多基于纯文本输入生成整体式三维模型。当文本描述包含多个物体的复杂场景时，这种方法会产生问题，因为向量化的文本嵌入本质上无法捕捉包含多实体与关系的复杂描述。对整个场景进行整体式三维建模进一步阻碍了文本实体与概念的精确锚定。为解决这一局限性，我们提出GraphDreamer——一种从场景图生成组合式三维场景的新框架，其中物体以节点表示，物体间的交互以边表示。通过利用场景图中的节点与边信息，我们的方法能更有效地运用预训练的文本到图像扩散模型，且无需图像级监督即可完全解耦不同物体。为便于建模物体间关系，我们采用符号距离场作为表示，并施加约束以避免物体间的相互穿透。为避免手动创建场景图，我们设计了一个针对ChatGPT的文本提示，使其能根据文本输入生成场景图。我们进行了定性与定量实验，验证了GraphDreamer在生成具有解耦物体实体的高保真组合式三维场景方面的有效性。