The advent of large language models, enabling flexibility through instruction-driven approaches, has revolutionized many traditional generative tasks, but large models for 3D data, particularly in comprehensively handling 3D shapes with other modalities, are still under-explored. By achieving instruction-based shape generations, versatile multimodal generative shape models can significantly benefit various fields like 3D virtual construction and network-aided design. In this work, we present ShapeGPT, a shape-included multi-modal framework to leverage strong pre-trained language models to address multiple shape-relevant tasks. Specifically, ShapeGPT employs a word-sentence-paragraph framework to discretize continuous shapes into shape words, further assembles these words for shape sentences, as well as integrates shape with instructional text for multi-modal paragraphs. To learn this shape-language model, we use a three-stage training scheme, including shape representation, multimodal alignment, and instruction-based generation, to align shape-language codebooks and learn the intricate correlations among these modalities. Extensive experiments demonstrate that ShapeGPT achieves comparable performance across shape-relevant tasks, including text-to-shape, shape-to-text, shape completion, and shape editing.

翻译：大型语言模型的出现，通过指令驱动方法实现了灵活性，彻底改变了众多传统生成任务，但面向3D数据的大型模型，特别是在全面处理3D形状与其他模态结合方面，仍处于探索不足的状态。通过实现基于指令的形状生成，通用的多模态生成形状模型可显著惠及3D虚拟构建和网络辅助设计等多个领域。本文提出ShapeGPT，一种包含形状的多模态框架，利用强大的预训练语言模型处理多项形状相关任务。具体而言，ShapeGPT采用词-句-段框架，将连续形状离散化为形状词，进一步组合这些词形成形状句，并将形状与指令文本集成形成多模态段落。为学习这一形状-语言模型，我们采用三阶段训练方案，包括形状表征、多模态对齐和基于指令的生成，以对齐形状-语言码本并学习这些模态间的复杂关联。大量实验表明，ShapeGPT在包括文本到形状、形状到文本、形状补全和形状编辑等形状相关任务上取得了可比性能。