Neural implicit fields have emerged as a powerful 3D representation for reconstructing and rendering photo-realistic views, yet they possess limited editability. Conversely, explicit 3D representations, such as polygonal meshes, offer ease of editing but may not be as suitable for rendering high-quality novel views. To harness the strengths of both representations, we propose a new approach that employs a mesh as a guiding mechanism in editing the neural radiance field. We first introduce a differentiable method using marching tetrahedra for polygonal mesh extraction from the neural implicit field and then design a differentiable color extractor to assign colors obtained from the volume renderings to this extracted mesh. This differentiable colored mesh allows gradient back-propagation from the explicit mesh to the implicit fields, empowering users to easily manipulate the geometry and color of neural implicit fields. To enhance user control from coarse-grained to fine-grained levels, we introduce an octree-based structure into its optimization. This structure prioritizes the edited regions and the surface part, making our method achieve fine-grained edits to the neural implicit field and accommodate various user modifications, including object additions, component removals, specific area deformations, and adjustments to local and global colors. Through extensive experiments involving diverse scenes and editing operations, we have demonstrated the capabilities and effectiveness of our method. Our project page is: \url{https://cassiepython.github.io/MNeuEdit/}
翻译:神经隐式场已成为一种强大的三维表示,用于重建和渲染逼真视图,但其编辑能力有限。相反,多边形网格等显式三维表示易于编辑,但可能不适合渲染高质量的新视图。为发挥两种表示的优势,我们提出一种新方法,利用网格作为引导机制来编辑神经辐射场。我们首先引入一种可微方法,使用行进四面体从神经隐式场提取多边形网格,然后设计一种可微颜色提取器,将从体渲染中获得的颜色分配给提取的网格。这种可微彩色网格允许梯度从显式网格反向传播到隐式场,使用户能够轻松操纵神经隐式场的几何形状和颜色。为增强从粗粒度到细粒度的用户控制,我们在其优化中引入基于八叉树的结构。该结构优先处理编辑区域和表面部分,使我们的方法能够对神经隐式场进行细粒度编辑,并适应各种用户修改,包括添加物体、移除组件、特定区域变形以及调整局部和全局颜色。通过涉及不同场景和编辑操作的大量实验,我们展示了该方法的能力和有效性。我们的项目页面为:\url{https://cassiepython.github.io/MNeuEdit/}