With the help of Score Distillation Sampling (SDS) and the rapid development of neural 3D representations, some methods have been proposed to perform 3D editing such as adding additional geometries, or overwriting textures. However, generalized 3D non-rigid editing task, which requires changing both the structure (posture or composition) and appearance (texture) of the original object, remains to be challenging in 3D editing field. In this paper, we propose Plasticine3D, a novel text-guided fine-grained controlled 3D editing pipeline that can perform 3D non-rigid editing with large structure deformations. Our work divides the editing process into a geometry editing stage and a texture editing stage to achieve separate control of structure and appearance. In order to maintain the details of the original object from different viewpoints, we propose a Multi-View-Embedding (MVE) Optimization strategy to ensure that the guidance model learns the features of the original object from various viewpoints. For the purpose of fine-grained control, we propose Embedding-Fusion (EF) to blend the original characteristics with the editing objectives in the embedding space, and control the extent of editing by adjusting the fusion rate. Furthermore, in order to address the issue of gradual loss of details during the generation process under high editing intensity, as well as the problem of insignificant editing effects in some scenarios, we propose Score Projection Sampling (SPS) as a replacement of score distillation sampling, which introduces additional optimization phases for editing target enhancement and original detail maintenance, leading to better editing quality. Extensive experiments demonstrate the effectiveness of our method on 3D non-rigid editing tasks
翻译:借助评分蒸馏采样(SDS)与神经三维表征的快速发展,已有方法能够实现诸如添加几何结构或覆盖纹理等三维编辑任务。然而,广义的三维非刚性编辑任务——需要同时改变原始物体的结构(姿态或构成)与外观(纹理)——在三维编辑领域仍具挑战性。本文提出Plasticine3D,一种新颖的文本引导细粒度可控三维编辑流程,能够实现具有大幅结构变形的三维非刚性编辑。我们的工作将编辑过程分解为几何编辑阶段与纹理编辑阶段,以实现结构与外观的分离控制。为从不同视角保持原始物体的细节,我们提出多视角嵌入(MVE)优化策略,确保引导模型从多角度学习原始物体的特征。为实现细粒度控制,我们提出嵌入融合(EF)方法,在嵌入空间中将原始特征与编辑目标进行混合,并通过调整融合率控制编辑程度。此外,为解决高编辑强度下生成过程中细节逐渐丢失的问题,以及部分场景中编辑效果不显著的问题,我们提出评分投影采样(SPS)以替代评分蒸馏采样,该方法引入了针对编辑目标增强与原始细节保持的额外优化阶段,从而获得更优的编辑质量。大量实验证明了我们的方法在三维非刚性编辑任务上的有效性。