Fine-tuning is an essential process to improve the performance of Large Language Models (LLMs) in specific domains, with Parameter-Efficient Fine-Tuning (PEFT) gaining popularity due to its capacity to reduce computational demands through the integration of low-rank adapters. These lightweight adapters, such as LoRA, can be shared and utilized on open-source platforms. However, adversaries could exploit this mechanism to inject backdoors into these adapters, resulting in malicious behaviors like incorrect or harmful outputs, which pose serious security risks to the community. Unfortunately, few of the current efforts concentrate on analyzing the backdoor patterns or detecting the backdoors in the adapters. To fill this gap, we first construct (and will release) PADBench, a comprehensive benchmark that contains 13,300 benign and backdoored adapters fine-tuned with various datasets, attack strategies, PEFT methods, and LLMs. Moreover, we propose PEFTGuard, the first backdoor detection framework against PEFT-based adapters. Extensive evaluation upon PADBench shows that PEFTGuard outperforms existing detection methods, achieving nearly perfect detection accuracy (100%) in most cases. Notably, PEFTGuard exhibits zero-shot transferability on three aspects, including different attacks, PEFT methods, and adapter ranks. In addition, we consider various adaptive attacks to demonstrate the high robustness of PEFTGuard. We further explore several possible backdoor mitigation defenses, finding fine-mixing to be the most effective method. We envision our benchmark and method can shed light on future LLM backdoor detection research.

翻译：微调是提升大型语言模型在特定领域性能的关键过程，其中参数高效微调因其通过集成低秩适配器降低计算需求的优势而日益普及。诸如LoRA的轻量级适配器可在开源平台上共享使用。然而，攻击者可能利用此机制向适配器注入后门，导致产生错误或有害输出等恶意行为，对社区构成严重安全威胁。遗憾的是，现有研究鲜少关注适配器中后门模式的分析或检测。为填补这一空白，我们首先构建（并将开源）PADBench——一个包含13,300个良性及后门适配器的综合基准，这些适配器通过多样化数据集、攻击策略、PEFT方法及LLM进行微调。此外，我们提出首个针对基于PEFT适配器的后门检测框架PEFTGuard。在PADBench上的大量实验表明，PEFTGuard显著优于现有检测方法，在多数情况下实现近乎完美的检测准确率（100%）。值得注意的是，PEFTGuard在攻击方式、PEFT方法和适配器秩三个维度上均展现出零样本迁移能力。我们还通过多种自适应攻击验证了PEFTGuard的高鲁棒性。进一步探索后门缓解防御方案时，发现精细混合策略最为有效。我们期望本基准与方法能为未来LLM后门检测研究提供重要参考。