With the development of customized large language model (LLM) agents, a new threat of black-box backdoor attacks has emerged, where malicious instructions are injected into hidden system prompts. These attacks easily bypass existing defenses that rely on white-box access, posing a serious security challenge. To address this, we propose SLIP, a Soft Label mechanism and key-extraction-guided CoT-based defense against Instruction backdoors in APIs. SLIP is designed based on two key insights. First, to counteract the model's oversensitivity to triggers, we propose a Key-extraction-guided Chain-of-Thought (KCoT). Instead of only considering the single trigger or the input sentence, KCoT prompts the agent to extract task-relevant key phrases. Second, to guide the LLM toward correct answers, our proposed Soft Label Mechanism (SLM) prompts the agent to quantify the semantic correlation between key phrases and candidate answers. Crucially, to mitigate the influence of residual triggers or misleading content in phrases extracted by KCoT, which typically causes anomalous scores, SLM excludes anomalous scores deviating significantly from the mean and subsequently averages the remaining scores to derive a more reliable semantic representation. Extensive experiments on classification and question-answer (QA) tasks demonstrate that SLIP is highly effective, reducing the average attack success rate (ASR) from 90.2% to 25.13% while maintaining high accuracy on clean data and outperforming state-of-the-art defenses. Our code are available in https://github.com/CAU-ISS-Lab/Backdoor-Attack-Defense-LLMs/tree/main/SLIP.

翻译：随着定制化大语言模型（LLM）智能体的发展，一种新型黑盒后门攻击威胁逐渐显现，即恶意指令被注入隐藏的系统提示中。此类攻击能够轻易绕过依赖白盒访问的现有防御机制，构成严峻的安全挑战。为此，我们提出SLIP——一种针对API中指令后门的软标签机制与关键提取引导链式思维（CoT）的防御方法。SLIP的设计基于两个关键洞见。首先，为抵消模型对触发词的过度敏感性，我们提出关键提取引导链式思维（KCoT）。KCoT不再仅关注单一触发词或输入句子，而是引导智能体提取与任务相关的关键短语。其次，为引导LLM生成正确答案，我们提出的软标签机制（SLM）促使智能体量化关键短语与候选答案之间的语义相关性。关键在于，为减轻KCoT提取的短语中残留触发词或误导性内容的影响（这类内容通常导致异常评分），SLM会排除显著偏离平均值的异常评分，随后对剩余评分进行平均，从而获得更可靠的语义表征。在分类与问答（QA）任务上的大量实验表明，SLIP具有高度有效性，能将平均攻击成功率（ASR）从90.2%降至25.13%，同时在干净数据上保持高准确率，且性能优于现有最先进的防御方法。我们的代码公开于：https://github.com/CAU-ISS-Lab/Backdoor-Attack-Defense-LLMs/tree/main/SLIP。