As large language models (LLMs) gain popularity, their vulnerability to adversarial attacks emerges as a primary concern. While fine-tuning models on domain-specific datasets is often employed to improve model performance, it can inadvertently introduce vulnerabilities within the underlying model. In this work, we investigate Accidental Vulnerability, unexpected vulnerabilities arising from characteristics of fine-tuning data. We begin by identifying potential correlation factors such as linguistic features, semantic similarity, and toxicity across multiple experimental datasets. We then evaluate the adversarial robustness of these fine-tuned models, analyzing persona shifts and interpretability traits to understand how dataset factors contribute to attack success rates. Lastly, we explore causal relationships that offer new insights into adversarial defense strategies, highlighting the crucial role of dataset design in preserving model alignment. Our code is available at https://github.com/psyonp/accidental_vulnerability.

翻译：随着大型语言模型（LLMs）的普及，其对抗攻击的脆弱性已成为主要关注点。尽管在特定领域数据集上微调模型常被用于提升模型性能，但这一过程可能无意中在基础模型中引入漏洞。本研究探讨了“意外漏洞”——即由微调数据特性引发的非预期脆弱性。我们首先在多个实验数据集中识别潜在相关因素，如语言特征、语义相似性和毒性。随后评估这些微调模型的对抗鲁棒性，通过分析角色偏移和可解释性特征来理解数据集因素如何影响攻击成功率。最后，我们探索了能够为对抗防御策略提供新见解的因果关系，强调数据集设计在保持模型对齐性中的关键作用。代码发布于 https://github.com/psyonp/accidental_vulnerability。