We propose 3Deformer, a general-purpose framework for interactive 3D shape editing. Given a source 3D mesh with semantic materials, and a user-specified semantic image, 3Deformer can accurately edit the source mesh following the shape guidance of the semantic image, while preserving the source topology as rigid as possible. Recent studies of 3D shape editing mostly focus on learning neural networks to predict 3D shapes, which requires high-cost 3D training datasets and is limited to handling objects involved in the datasets. Unlike these studies, our 3Deformer is a non-training and common framework, which only requires supervision of readily-available semantic images, and is compatible with editing various objects unlimited by datasets. In 3Deformer, the source mesh is deformed utilizing the differentiable renderer technique, according to the correspondences between semantic images and mesh materials. However, guiding complex 3D shapes with a simple 2D image incurs extra challenges, that is, the deform accuracy, surface smoothness, geometric rigidity, and global synchronization of the edited mesh should be guaranteed. To address these challenges, we propose a hierarchical optimization architecture to balance the global and local shape features, and propose further various strategies and losses to improve properties of accuracy, smoothness, rigidity, and so on. Extensive experiments show that our 3Deformer is able to produce impressive results and reaches the state-of-the-art level.

翻译：我们提出3Deformer，这是一个用于交互式三维形状编辑的通用框架。给定一个具有语义材质的源三维网格，以及用户指定的语义图像，3Deformer能够按照语义图像的形状引导精确编辑源网格，同时尽可能刚性地保持源拓扑结构。近年来的三维形状编辑研究主要集中于学习神经网络以预测三维形状，这需要高成本的三维训练数据集，并且仅限于处理数据集中涉及的物体。与这些研究不同，我们的3Deformer是一种无需训练的通用框架，仅需利用易于获取的语义图像作为监督，并且兼容编辑不受数据集限制的各种物体。在3Deformer中，源网格依据语义图像与网格材质之间的对应关系，利用可微分渲染器技术进行变形。然而，用简单的二维图像引导复杂的三维形状会带来额外挑战，即需保证编辑后网格的变形精度、表面平滑度、几何刚性和全局同步性。为解决这些挑战，我们提出一种分层优化架构来平衡全局与局部形状特征，并进一步提出多种策略和损失函数以提升精度、平滑度、刚性等性能。大量实验表明，我们的3Deformer能够生成令人印象深刻的结果，并达到当前最优水平。