Recent works have demonstrated that natural language can be used to generate and edit 3D shapes. However, these methods generate shapes with limited fidelity and diversity. We introduce CLIP-Sculptor, a method to address these constraints by producing high-fidelity and diverse 3D shapes without the need for (text, shape) pairs during training. CLIP-Sculptor achieves this in a multi-resolution approach that first generates in a low-dimensional latent space and then upscales to a higher resolution for improved shape fidelity. For improved shape diversity, we use a discrete latent space which is modeled using a transformer conditioned on CLIP's image-text embedding space. We also present a novel variant of classifier-free guidance, which improves the accuracy-diversity trade-off. Finally, we perform extensive experiments demonstrating that CLIP-Sculptor outperforms state-of-the-art baselines. The code is available at https://ivl.cs.brown.edu/#/projects/clip-sculptor.

翻译：近期研究已表明，自然语言可用于生成和编辑三维形状。然而，现有方法生成的形状在逼真度和多样性方面存在局限。我们提出CLIP-Sculptor方法，该方法无需训练过程中的（文本，形状）配对数据，即可生成高保真且多样化的三维形状。CLIP-Sculptor采用多分辨率方法实现这一目标：首先生成低维潜空间中的形状，随后将其上采样至更高分辨率以提升形状保真度。为增强形状多样性，我们使用由变压器网络建模的离散潜空间，该网络以CLIP的图文联合嵌入空间为条件。此外，我们提出一种无分类器引导的新变体，可改善准确率与多样性之间的权衡。最后，通过大量实验证明，CLIP-Sculptor在性能上优于当前最先进的基线方法。代码已开源：https://ivl.cs.brown.edu/#/projects/clip-sculptor。