With the emergence of more powerful large language models (LLMs), such as ChatGPT and GPT-4, in-context learning (ICL) has gained significant prominence in leveraging these models for specific tasks by utilizing data-label pairs as precondition prompts. While incorporating demonstrations can greatly enhance the performance of LLMs across various tasks, it may introduce a new security concern: attackers can manipulate only the demonstrations without changing the input to perform an attack. In this paper, we investigate the security concern of ICL from an adversarial perspective, focusing on the impact of demonstrations. We propose an ICL attack based on TextAttack, which aims to only manipulate the demonstration without changing the input to mislead the models. Our results demonstrate that as the number of demonstrations increases, the robustness of in-context learning would decreases. Furthermore, we also observe that adversarially attacked demonstrations exhibit transferability to diverse input examples. These findings emphasize the critical security risks associated with ICL and underscore the necessity for extensive research on the robustness of ICL, particularly given its increasing significance in the advancement of LLMs.

翻译：随着ChatGPT和GPT-4等更强大的大型语言模型（LLMs）的出现，上下文学习（ICL）通过利用数据-标签对作为前提提示，在利用这些模型完成特定任务方面获得了显著重要性。虽然包含演示可以极大地提升LLMs在各种任务上的性能，但这可能引入新的安全隐忧：攻击者可以仅操纵演示而不改变输入来发起攻击。在本文中，我们从对抗性角度研究ICL的安全问题，重点关注演示的影响。我们提出了一种基于TextAttack的ICL攻击方法，旨在仅操纵演示而不改变输入，从而误导模型。研究结果表明，随着演示数量的增加，上下文学习的鲁棒性会下降。此外，我们还观察到，受对抗性攻击的演示能够迁移到不同的输入示例中。这些发现强调了与ICL相关的关键安全风险，并凸显了对ICL鲁棒性进行广泛研究的必要性，尤其是在其对于LLMs发展日益重要的背景下。