Training or finetuning large-scale language models (LLMs) requires substantial computation resources, motivating recent efforts to explore parameter-efficient adaptation to downstream tasks. One approach is to treat these models as black boxes and use forward passes (Inference APIs) to interact with them. Current research focuses on adapting these black-box models to downstream tasks using gradient-free prompt optimization, but this often involves an expensive process of searching task-specific prompts. Therefore, we are motivated to study black-box language model adaptation without prompt search. Specifically, we introduce a label-enhanced cross-attention network called CrossTune, which models the semantic relatedness between the input text sequence and task-specific label descriptions. Its effectiveness is examined in the context of few-shot text classification. To improve the generalization of CrossTune, we utilize ChatGPT to generate additional training data through in-context learning. A switch mechanism is implemented to exclude low-quality ChatGPT-generated data. Through extensive experiments on seven benchmark text classification datasets, we demonstrate that our proposed approach outperforms the previous state-of-the-art gradient-free black-box tuning method by 5.7% on average. Even without using ChatGPT-augmented data, CrossTune performs better or comparably than previous black-box tuning methods, suggesting the effectiveness of our approach.

翻译：大规模语言模型的训练或微调需要大量计算资源，这促使近年来的研究探索面向下游任务的参数高效适配方法。一种方法是将这些模型视为黑盒，通过前向传播（推理API）与其交互。当前研究聚焦于使用无梯度提示优化来适配这些黑盒模型至下游任务，但这一过程通常涉及针对特定任务进行代价高昂的提示搜索。因此，我们致力于研究无需提示搜索的黑盒语言模型适配方法。具体而言，我们提出了一种名为CrossTune的标签增强交叉注意力网络，该网络能够建模输入文本序列与任务特定标签描述之间的语义关联性，并在少样本文本分类任务中验证其有效性。为提升CrossTune的泛化性能，我们利用ChatGPT通过上下文学习生成额外训练数据，并设计切换机制以筛除低质量ChatGPT生成数据。通过在七个基准文本分类数据集上的大量实验，我们证明所提方法较先前最先进的无梯度黑盒调优方法平均提升5.7%。即使未使用ChatGPT增强数据，CrossTune的性能仍优于或持平于以往黑盒调优方法，这充分验证了我们方法的有效性。