Instruction-tuned Code Large Language Models (Code LLMs) are increasingly utilized as AI coding assistants and integrated into various applications. However, the cybersecurity vulnerabilities and implications arising from the widespread integration of these models are not yet fully understood due to limited research in this domain. To bridge this gap, this paper presents EvilInstructCoder, a framework specifically designed to assess the cybersecurity vulnerabilities of instruction-tuned Code LLMs to adversarial attacks. EvilInstructCoder introduces the Adversarial Code Injection Engine to automatically generate malicious code snippets and inject them into benign code to poison instruction tuning datasets. It incorporates practical threat models to reflect real-world adversaries with varying capabilities and evaluates the exploitability of instruction-tuned Code LLMs under these diverse adversarial attack scenarios. Through the use of EvilInstructCoder, we conduct a comprehensive investigation into the exploitability of instruction tuning for coding tasks using three state-of-the-art Code LLM models: CodeLlama, DeepSeek-Coder, and StarCoder2, under various adversarial attack scenarios. Our experimental results reveal a significant vulnerability in these models, demonstrating that adversaries can manipulate the models to generate malicious payloads within benign code contexts in response to natural language instructions. For instance, under the backdoor attack setting, by poisoning only 81 samples (0.5\% of the entire instruction dataset), we achieve Attack Success Rate at 1 (ASR@1) scores ranging from 76\% to 86\% for different model families. Our study sheds light on the critical cybersecurity vulnerabilities posed by instruction-tuned Code LLMs and emphasizes the urgent necessity for robust defense mechanisms to mitigate the identified vulnerabilities.

翻译：指令微调代码大语言模型（Code LLMs）正日益被用作AI编程助手并集成到各类应用中。然而，由于该领域研究的局限性，这些模型广泛集成所带来的网络安全漏洞及潜在影响尚未被充分理解。为填补这一空白，本文提出了EvilInstructCoder框架，该框架专门用于评估经过指令微调的Code LLMs在对抗攻击下的网络安全漏洞。EvilInstructCoder引入了对抗性代码注入引擎，可自动生成恶意代码片段并将其注入良性代码中，以污染指令微调数据集。它结合了实用的威胁模型以反映现实世界中具备不同能力的对抗者，并评估了在这些多样化对抗攻击场景下指令微调Code LLMs的可利用性。通过使用EvilInstructCoder，我们利用三种最先进的Code LLM模型——CodeLlama、DeepSeek-Coder和StarCoder2——在不同的对抗攻击场景下，对编码任务中指令微调的可利用性进行了全面研究。实验结果表明，这些模型存在显著漏洞，表明对抗者能够操纵模型在自然语言指令的触发下，在良性代码上下文中生成恶意载荷。例如，在后门攻击设置下，仅污染81个样本（占整个指令数据集的0.5%），我们在不同模型家族上就获得了攻击成功率（ASR@1）得分从76%到86%不等。我们的研究揭示了指令微调Code LLMs所引发的关键网络安全漏洞，并强调了迫切需要强大的防御机制来缓解已识别的漏洞。