Score distillation sampling (SDS) has emerged as an effective framework in text-driven 3D editing tasks, leveraging diffusion models for 3D-consistent editing. However, existing SDS-based 3D editing methods suffer from long training times and produce low-quality results. We identify that the root cause of this performance degradation is \textit{their conflict with the sampling dynamics of diffusion models}. Addressing this conflict allows us to treat SDS as a diffusion reverse process for 3D editing via sampling from data space. In contrast, existing methods naively distill the score function using diffusion models. From these insights, we propose DreamCatalyst, a novel framework that considers these sampling dynamics in the SDS framework. Specifically, we devise the optimization process of our DreamCatalyst to approximate the diffusion reverse process in editing tasks, thereby aligning with diffusion sampling dynamics. As a result, DreamCatalyst successfully reduces training time and improves editing quality. Our method offers two modes: (1) a fast mode that edits Neural Radiance Fields (NeRF) scenes approximately 23 times faster than current state-of-the-art NeRF editing methods, and (2) a high-quality mode that produces superior results about 8 times faster than these methods. Notably, our high-quality mode outperforms current state-of-the-art NeRF editing methods in terms of both speed and quality. DreamCatalyst also surpasses the state-of-the-art 3D Gaussian Splatting (3DGS) editing methods, establishing itself as an effective and model-agnostic 3D editing solution. See more extensive results on our project page: https://dream-catalyst.github.io.
翻译:分数蒸馏采样(SDS)已成为文本驱动三维编辑任务中的有效框架,其利用扩散模型实现三维一致性编辑。然而,现有基于SDS的三维编辑方法存在训练时间长、生成质量低的问题。我们发现其性能下降的根本原因在于**与扩散模型的采样动力学相冲突**。解决这一冲突使我们能够将SDS视为通过从数据空间采样进行三维编辑的扩散逆向过程。相比之下,现有方法仅简单地利用扩散模型蒸馏分数函数。基于这些洞见,我们提出了DreamCatalyst——一个在SDS框架中充分考虑采样动力学的新型框架。具体而言,我们设计了DreamCatalyst的优化过程以近似编辑任务中的扩散逆向过程,从而与扩散采样动力学保持一致。因此,DreamCatalyst成功缩短了训练时间并提升了编辑质量。我们的方法提供两种模式:(1)快速模式,其编辑神经辐射场(NeRF)场景的速度比当前最先进的NeRF编辑方法快约23倍;(2)高质量模式,其以比这些方法快约8倍的速度生成更优的结果。值得注意的是,我们的高质量模式在速度和质量上均优于当前最先进的NeRF编辑方法。DreamCatalyst亦超越了最先进的三维高斯溅射(3DGS)编辑方法,确立了其作为有效且模型无关的三维编辑解决方案的地位。更多详尽结果请参见项目页面:https://dream-catalyst.github.io。