Direct mesh editing and deformation are key components in the geometric modeling and animation pipeline. Direct mesh editing methods are typically framed as optimization problems combining user-specified vertex constraints with a regularizer that determines the position of the rest of the vertices. The choice of the regularizer is key to the realism and authenticity of the final result. Physics and geometry-based regularizers are not aware of the global context and semantics of the object, and the more recent deep learning priors are limited to a specific class of 3D object deformations. In this work, our main contribution is a local mesh editing method called DragD3D for global context-aware realistic deformation through direct manipulation of a few vertices. DragD3D is not restricted to any class of objects. It achieves this by combining the classic geometric ARAP (as rigid as possible) regularizer with 2D priors obtained from a large-scale diffusion model. Specifically, we render the objects from multiple viewpoints through a differentiable renderer and use the recently introduced DDS loss which scores the faithfulness of the rendered image to one from a diffusion model. DragD3D combines the approximate gradients of the DDS with gradients from the ARAP loss to modify the mesh vertices via neural Jacobian field, while also satisfying vertex constraints. We show that our deformations are realistic and aware of the global context of the objects, and provide better results than just using geometric regularizers.

翻译：直接网格编辑与变形是几何建模及动画流程中的关键组成部分。直接网格编辑方法通常被构建为优化问题，将用户指定的顶点约束与决定其余顶点位置的规整器相结合。规整器的选择对于最终结果的真实性与可信度至关重要。基于物理和几何的规整器无法感知物体的全局上下文与语义，而较新的深度学习先验则局限于特定类别的三维物体变形。本文的主要贡献在于提出一种名为DragD3D的局部网格编辑方法，该方法通过直接操作少量顶点实现具有全局上下文感知的真实变形。DragD3D不局限于任何物体类别。其通过将经典的几何ARAP（尽可能刚性）规整器与从大规模扩散模型中获取的二维先验相结合来实现这一目标。具体而言，我们利用可微渲染器从多个视角渲染物体，并采用近期提出的DDS损失函数，该函数衡量渲染图像与扩散模型生成图像之间的保真度。DragD3D将DDS的近似梯度与ARAP损失的梯度相结合，通过神经雅可比场修改网格顶点，同时满足顶点约束。我们证明，本方法的变形结果具有真实性，且能感知物体的全局上下文，相比仅使用几何规整器取得了更优的效果。