Text-to-3D generation has achieved significant success by incorporating powerful 2D diffusion models, but insufficient 3D prior knowledge also leads to the inconsistency of 3D geometry. Recently, since large-scale multi-view datasets have been released, fine-tuning the diffusion model on the multi-view datasets becomes a mainstream to solve the 3D inconsistency problem. However, it has confronted with fundamental difficulties regarding the limited quality and diversity of 3D data, compared with 2D data. To sidestep these trade-offs, we explore a retrieval-augmented approach tailored for score distillation, dubbed ReDream. We postulate that both expressiveness of 2D diffusion models and geometric consistency of 3D assets can be fully leveraged by employing the semantically relevant assets directly within the optimization process. To this end, we introduce novel framework for retrieval-based quality enhancement in text-to-3D generation. We leverage the retrieved asset to incorporate its geometric prior in the variational objective and adapt the diffusion model's 2D prior toward view consistency, achieving drastic improvements in both geometry and fidelity of generated scenes. We conduct extensive experiments to demonstrate that ReDream exhibits superior quality with increased geometric consistency. Project page is available at https://ku-cvlab.github.io/ReDream/.

翻译：文本生成3D通过引入强大的2D扩散模型取得了显著成功，但缺乏3D先验知识导致三维几何不一致性问题。近期，随着大规模多视图数据集的发布，在多视图数据集上微调扩散模型成为解决3D不一致性问题的主流方法。然而，该方法面临根本性困难：相较于2D数据，3D数据在质量和多样性方面存在局限性。为规避这些权衡，我们探索了一种专门针对得分蒸馏的检索增强方法，命名为ReDream。我们假设通过直接在优化过程中利用语义相关的3D资产，可以充分发挥2D扩散模型的表达能力和3D资产几何一致性。为此，我们提出了一种新颖的基于检索的文本生成3D质量增强框架。我们利用检索到的资产将几何先验整合到变分目标中，并调整扩散模型的2D先验以增强视图一致性，从而在生成场景的几何结构和保真度方面实现显著改进。大量实验表明，ReDream在保持更高几何一致性的同时展现出卓越的生成质量。项目页面访问地址：https://ku-cvlab.github.io/ReDream/。