Large Language Models have demonstrated significant ability in accomplishing a wide range of Natural Language Processing (NLP) tasks. However, their performance is highly sensitive to the even minor changes in the phrasing of the task instructions, leading to a line of research in automatic instruction optimization towards better performance for NLP tasks. Unfortunately, existing methods for instruction optimization fail to consider the distribution shift between the seen training data and the unseen test data, where testing on unseen group of data with a different distribution could potentially lead to performance drop. In this paper, we take an initial step of investigating the problem of LLM instruction optimization across data groups with distribution shifts. We find that the optimal instructions do encounter performance drops on LLM under certain distribution shifts. To this end, we propose a framework to derive more robust optimal instructions that improve the performance on the unseen data group without large sacrifice on the seen data group. Experimental results demonstrate the effectiveness of our proposed framework.

翻译：大语言模型在完成各类自然语言处理任务中展现出显著能力。然而，其性能对任务指令措辞的细微变化高度敏感，由此催生了针对自动优化指令以提升自然语言处理任务性能的研究方向。遗憾的是，现有指令优化方法未能考虑已见训练数据与未见测试数据之间的分布偏移——当对具有不同分布的未见数据组进行测试时，可能导致性能下降。本文首次探索了分布偏移下数据组间大语言模型指令优化问题。我们发现，最优指令在特定分布偏移条件下确实会导致大语言模型性能下降。为此，我们提出一个框架以推导更鲁棒的最优指令，在不大幅牺牲已见数据组性能的同时提升未见数据组的表现。实验结果验证了所提框架的有效性。