Parameter-efficient fine-tuning (PEFT) methods have provided an effective way for adapting large vision-language models to specific tasks or scenarios. Typically, they learn a very small scale of parameters for pre-trained models in a white-box formulation, which assumes model architectures to be known and parameters to be accessible. However, large models are often not open-source due to considerations of preventing abuse or commercial factors, hence posing a barrier to the deployment of white-box PEFT methods. To alleviate the dependence on model accessibility, we introduce collaborative black-box tuning (CBBT) for both textual prompt optimization and output feature adaptation for black-box models. Specifically, considering that the backpropagation gradients are blocked, we approximate the gradients of textual prompts by analyzing the predictions with perturbed prompts. Secondly, a lightweight adapter is deployed over the output feature of the inaccessible model, further facilitating the model adaptation process. Empowered with these designs, our CBBT is extensively evaluated on eleven downstream benchmarks and achieves remarkable improvements compared to existing black-box VL adaptation methods. Code is released at https://github.com/guozix/cbbt.

翻译：参数高效微调（PEFT）方法为将大型视觉语言模型适配到特定任务或场景提供了有效途径。通常，这些方法在白箱框架下学习预训练模型中极小规模的参数，要求已知模型架构且可访问参数。然而，受制于防止滥用或商业因素等考量，大型模型往往不开源，这为白箱PEFT方法的部署设置了障碍。为降低对模型可访问性的依赖，我们提出了协同黑箱调优（CBBT），针对黑箱模型同时实现文本提示优化与输出特征适配。具体而言，考虑到反向传播梯度被阻断，我们通过扰动提示的预测分析来近似文本提示的梯度。其次，在不可访问模型的输出特征上部署轻量级适配器，进一步促进模型适配过程。凭借这些设计，我们的CBBT在十一个下游基准上进行了广泛评估，相较于现有黑箱视觉语言适配方法取得了显著提升。代码已发布在https://github.com/guozix/cbbt。