The popularity of pre-trained large models has revolutionized downstream tasks across diverse fields, such as language, vision, and multi-modality. To minimize the adaption cost for downstream tasks, many Parameter-Efficient Fine-Tuning (PEFT) techniques are proposed for language and 2D image pre-trained models. However, the specialized PEFT method for 3D pre-trained models is still under-explored. To this end, we introduce Point-PEFT, a novel framework for adapting point cloud pre-trained models with minimal learnable parameters. Specifically, for a pre-trained 3D model, we freeze most of its parameters, and only tune the newly added PEFT modules on downstream tasks, which consist of a Point-prior Prompt and a Geometry-aware Adapter. The Point-prior Prompt adopts a set of learnable prompt tokens, for which we propose to construct a memory bank with domain-specific knowledge, and utilize a parameter-free attention to enhance the prompt tokens. The Geometry-aware Adapter aims to aggregate point cloud features within spatial neighborhoods to capture fine-grained geometric information through local interactions. Extensive experiments indicate that our Point-PEFT can achieve better performance than the full fine-tuning on various downstream tasks, while using only 5% of the trainable parameters, demonstrating the efficiency and effectiveness of our approach. Code will be released at https://github.com/Even-JK/PEFT-3D.

翻译：预训练大模型的普及革新了语言、视觉及多模态等各类下游任务。为减少下游任务的适配成本，研究者已提出多种针对语言和2D图像预训练模型的参数高效微调（PEFT）技术。然而，专门面向3D预训练模型的PEFT方法仍处于探索不足的状态。为此，我们提出Point-PEFT——一种以最少可学习参数适配点云预训练模型的新型框架。具体而言，对于预训练的3D模型，我们冻结其大部分参数，仅在下游任务中微调新增的PEFT模块，该模块由点先验提示（Point-prior Prompt）与几何感知适配器（Geometry-aware Adapter）组成。点先验提示采用一组可学习提示令牌，我们为其构建包含领域特定知识的记忆库，并利用无参数注意力机制增强提示令牌。几何感知适配器旨在通过空间邻域内的局部交互聚合点云特征，以捕获细粒度几何信息。大量实验表明，Point-PEFT在各类下游任务中性能优于全参数微调，同时仅使用5%的可训练参数，充分证明了该方法的高效性与有效性。代码将发布于 https://github.com/Even-JK/PEFT-3D。