The performance of large language models (LLMs) is acutely sensitive to the phrasing of prompts, which raises significant concerns about their reliability in real-world scenarios. Existing studies often divide prompts into task-level instructions and case-level inputs and primarily focus on evaluating and improving robustness against variations in tasks-level instructions. However, this setup fails to fully address the diversity of real-world user queries and assumes the existence of task-specific datasets. To address these limitations, we introduce RobustAlpacaEval, a new benchmark that consists of semantically equivalent case-level queries and emphasizes the importance of using the worst prompt performance to gauge the lower bound of model performance. Extensive experiments on RobustAlpacaEval with ChatGPT and six open-source LLMs from the Llama, Mistral, and Gemma families uncover substantial variability in model performance; for instance, a difference of 45.48% between the worst and best performance for the Llama-2-70B-chat model, with its worst performance dipping as low as 9.38%. We further illustrate the difficulty in identifying the worst prompt from both model-agnostic and model-dependent perspectives, emphasizing the absence of a shortcut to characterize the worst prompt. We also attempt to enhance the worst prompt performance using existing prompt engineering and prompt consistency methods, but find that their impact is limited. These findings underscore the need to create more resilient LLMs that can maintain high performance across diverse prompts. Data and code are available at https://github.com/cbwbuaa/On-the-Worst-Prompt- Performance-of-LLMs.

翻译：大型语言模型（LLMs）的性能对提示的措辞极为敏感，这引发了对其在现实场景中可靠性的重大担忧。现有研究通常将提示划分为任务级指令和案例级输入，并主要关注评估和改进针对任务级指令变化的鲁棒性。然而，这种设置未能充分应对现实世界用户查询的多样性，并假设存在特定任务的数据集。为应对这些局限，我们引入了RobustAlpacaEval，这是一个由语义等效的案例级查询组成的新基准，并强调了使用最差提示性能来衡量模型性能下限的重要性。在RobustAlpacaEval上对ChatGPT以及来自Llama、Mistral和Gemma家族的六个开源LLMs进行的广泛实验揭示了模型性能的巨大变异性；例如，Llama-2-70B-chat模型的最差与最佳性能之间差异达45.48%，其最差性能低至9.38%。我们进一步从模型无关和模型依赖两个角度阐述了识别最差提示的困难，强调了不存在刻画最差提示的捷径。我们还尝试使用现有的提示工程和提示一致性方法来提升最差提示性能，但发现其效果有限。这些发现强调了需要创建更具韧性的LLMs，使其能够在多样化的提示下保持高性能。数据和代码可在 https://github.com/cbwbuaa/On-the-Worst-Prompt- Performance-of-LLMs 获取。