The scale of large pre-trained models (PTMs) poses significant challenges in adapting to downstream tasks due to the high optimization overhead and storage costs associated with full-parameter fine-tuning. To address this, many studies explore parameter-efficient tuning methods, also framed as "delta tuning", which updates only a small subset of parameters, known as "delta modules", while keeping the backbone model's parameters fixed. However, the practicality and flexibility of delta tuning have been limited due to existing implementations that directly modify the code of the backbone PTMs and hard-code specific delta tuning methods for each PTM. In this paper, we present OpenDelta, an open-source library that overcomes these limitations by providing a plug-and-play implementation of various delta tuning methods. Our novel techniques eliminate the need to modify the backbone PTMs' code, making OpenDelta compatible with different, even novel PTMs. OpenDelta is designed to be simple, modular, and extensible, providing a comprehensive platform for researchers and practitioners to adapt large PTMs efficiently.

翻译：大型预训练模型因其庞大的参数规模，在适配下游任务时面临显著挑战：全参数微调需要高昂的优化开销和存储成本。为此，大量研究探索参数高效微调方法，即"增量调整"（delta tuning）范式——仅更新被称为"增量模块"（delta modules）的少量参数，同时保持主干模型参数固定不变。然而，现有实现方案需要对主干预训练模型的代码进行直接修改，并将特定增量调整方法硬编码至各预训练模型中，严重制约了增量调整的实用性与灵活性。本文提出OpenDelta这一开源工具库，通过提供多种增量调整方法的即插即用实现，突破上述限制。我们提出的创新技术无需修改主干预训练模型的代码，使OpenDelta能够兼容不同乃至新型的预训练模型。本库遵循简洁、模块化、可扩展的设计原则，为研究人员和从业者高效适配大规模预训练模型提供了综合性平台。