Large Language Models (LLMs) have shown remarkable proficiency in following instructions, making them valuable in customer-facing applications. However, their impressive capabilities also raise concerns about the amplification of risks posed by adversarial instructions, which can be injected into the model input by third-party attackers to manipulate LLMs' original instructions and prompt unintended actions and content. Therefore, it is crucial to understand LLMs' ability to accurately discern which instructions to follow to ensure their safe deployment in real-world scenarios. In this paper, we propose a pioneering benchmark for automatically evaluating the robustness of instruction-following LLMs against adversarial instructions injected in the prompt. The objective of this benchmark is to quantify the extent to which LLMs are influenced by injected adversarial instructions and assess their ability to differentiate between these injected adversarial instructions and original user instructions. Through experiments conducted with state-of-the-art instruction-following LLMs, we uncover significant limitations in their robustness against adversarial instruction injection attacks. Furthermore, our findings indicate that prevalent instruction-tuned models are prone to being ``overfitted'' to follow any instruction phrase in the prompt without truly understanding which instructions should be followed. This highlights the need to address the challenge of training models to comprehend prompts instead of merely following instruction phrases and completing the text. The data and code can be found at \url{https://github.com/Leezekun/Adv-Instruct-Eval}.

翻译：大型语言模型（LLMs）在遵循指令方面展现出卓越能力，使其在面向客户的应用程序中具有重要价值。然而，其强大能力也引发了对抗性指令带来的风险放大问题——第三方攻击者可能将这种指令注入模型输入，从而操纵LLMs的原始指令，并诱导其产生非预期行为与内容。因此，理解LLMs准确辨别应遵循何种指令的能力，对于确保其在现实场景中的安全部署至关重要。本文提出一项开创性基准测试，可自动评估遵循指令的LLMs对注入至提示中的对抗性指令的鲁棒性。该基准旨在量化注入的对抗性指令对LLMs的影响程度，并评估其区分注入指令与原始用户指令的能力。通过对当前最先进的指令遵循型LLMs进行实验，我们发现其在对抗性指令注入攻击下的鲁棒性存在显著局限。此外，研究结果表明，主流的指令微调模型容易陷入"过拟合"——机械地遵循提示中的任意指令短语，而未能真正理解应遵循哪些指令。这突显出需解决训练模型理解提示含义的挑战，而非仅让其机械地遵循指令短语并完成文本补全。数据与代码可参见\url{https://github.com/Leezekun/Adv-Instruct-Eval}。