The recent developments in neural fields have brought phenomenal capabilities to the field of shape generation, but they lack crucial properties, such as incremental control - a fundamental requirement for artistic work. Triangular meshes, on the other hand, are the representation of choice for most geometry related tasks, offering efficiency and intuitive control, but do not lend themselves to neural optimization. To support downstream tasks, previous art typically proposes a two-step approach, where first a shape is generated using neural fields, and then a mesh is extracted for further processing. Instead, in this paper we introduce a hybrid approach that maintains both a mesh and a Signed Distance Field (SDF) representations consistently. Using this representation, we introduce MagicClay - an artist friendly tool for sculpting regions of a mesh according to textual prompts while keeping other regions untouched. Our framework carefully and efficiently balances consistency between the representations and regularizations in every step of the shape optimization; Relying on the mesh representation, we show how to render the SDF at higher resolutions and faster. In addition, we employ recent work in differentiable mesh reconstruction to adaptively allocate triangles in the mesh where required, as indicated by the SDF. Using an implemented prototype, we demonstrate superior generated geometry compared to the state-of-the-art, and novel consistent control, allowing sequential prompt-based edits to the same mesh for the first time.

翻译：近期神经场的发展为形状生成领域带来了非凡的能力，但其缺乏关键特性，例如增量控制——这是艺术创作的基本要求。另一方面，三角网格作为大多数几何相关任务的首选表示方法，具有高效性和直观可控性，却难以适用于神经优化。为支持下游任务，现有技术通常采用两步法：首先生成神经场构建形状，随后提取网格进行进一步处理。本文提出一种混合方法，能同时保持网格与符号距离场（SDF）表示的一致性。基于此表示，我们推出MagicClay——一款艺术家友好型工具，可根据文本提示对网格特定区域进行雕刻，同时保持其他区域不变。我们的框架在形状优化的每一步都精细高效地平衡了两种表示间的一致性及正则化约束；借助网格表示，我们展示了如何以更高分辨率和更快速度渲染SDF。此外，我们采用可微分网格重建的最新研究成果，根据SDF指示在所需区域自适应分配网格三角形。通过实现的原型系统，我们展示了相较于现有技术更优越的生成几何效果，以及新颖的一致性控制能力，首次实现了对同一网格基于提示的连续编辑。