Transfer learning is important for foundation models to adapt to downstream tasks. However, many foundation models are proprietary, so users must share their data with model owners to fine-tune the models, which is costly and raise privacy concerns. Moreover, fine-tuning large foundation models is computation-intensive and impractical for most downstream users. In this paper, we propose Offsite-Tuning, a privacy-preserving and efficient transfer learning framework that can adapt billion-parameter foundation models to downstream data without access to the full model. In offsite-tuning, the model owner sends a light-weight adapter and a lossy compressed emulator to the data owner, who then fine-tunes the adapter on the downstream data with the emulator's assistance. The fine-tuned adapter is then returned to the model owner, who plugs it into the full model to create an adapted foundation model. Offsite-tuning preserves both parties' privacy and is computationally more efficient than the existing fine-tuning methods that require access to the full model weights. We demonstrate the effectiveness of offsite-tuning on various large language and vision foundation models. Offsite-tuning can achieve comparable accuracy as full model fine-tuning while being privacy-preserving and efficient, achieving 6.5x speedup and 5.6x memory reduction. Code is available at https://github.com/mit-han-lab/offsite-tuning.

翻译：迁移学习对于基础模型适应下游任务至关重要。然而，许多基础模型具有专有性，用户需将数据分享给模型所有者才能微调模型，这既成本高昂又引发隐私顾虑。此外，微调大型基础模型计算量巨大，对大多数下游用户而言不切实际。本文提出离站微调（Offsite-Tuning）——一种保护隐私且高效的迁移学习框架，可在无需访问完整模型的情况下将十亿参数级别的基础模型适配至下游数据。在离站微调中，模型所有者向数据所有者发送轻量级适配器与有损压缩仿真器，数据所有者借助仿真器在下游数据上微调适配器。微调后的适配器返回模型所有者，后者将其接入完整模型以生成适配后的基础模型。离站微调保护双方隐私，且比现有需要访问完整模型权重的微调方法更高效。我们在多种大型语言与视觉基础模型上证明了离站微调的有效性。该方法可在保护隐私并提升效率的同时达到与完整模型微调接近的精度，实现6.5倍加速与5.6倍内存缩减。代码开源地址：https://github.com/mit-han-lab/offsite-tuning。