Benchmarking large language models (LLMs) is critical for understanding their capabilities, limitations, and robustness. In addition to interface artifacts, prior studies have shown that LLM decisions can be influenced by directive signals such as social cues, framing, and instructions. In this work, we introduce option injection, a benchmarking approach that augments the multiple-choice question answering (MCQA) interface with an additional option containing a misleading directive, leveraging standardized choice structure and scalable evaluation. We construct OI-Bench, a benchmark of 3,000 questions spanning knowledge, reasoning, and commonsense tasks, with 16 directive types covering social compliance, bonus framing, threat framing, and instructional interference. This setting combines manipulation of the choice interface with directive-based interference, enabling systematic assessment of model susceptibility. We evaluate 12 LLMs to analyze attack success rates, behavioral responses, and further investigate mitigation strategies ranging from inference-time prompting to post-training alignment. Experimental results reveal substantial vulnerabilities and heterogeneous robustness across models. OI-Bench is expected to support more systematic evaluation of LLM robustness to directive interference within choice-based interfaces.

翻译：对大语言模型（LLM）进行基准测试对于理解其能力、局限性和鲁棒性至关重要。除了界面人为因素外，先前研究表明，LLM的决策可能受到社会线索、框架效应和指令等导向性信号的干扰。本研究提出选项注入这一基准测试方法，该方法通过在多选题（MCQA）界面中增加一个包含误导性指令的选项，利用标准化的选项结构和可扩展的评估机制。我们构建了OI-Bench基准数据集，包含涵盖知识、推理和常识任务的3000个问题，并设计了16种指令类型，涉及社会顺从性、奖励框架、威胁框架和指令干扰。该设置将选项界面的操控与基于指令的干扰相结合，能够系统评估模型对干扰的敏感性。我们评估了12个LLM，分析了攻击成功率与行为响应模式，并进一步探究了从推理时提示到训练后对齐等多种缓解策略。实验结果表明，各模型存在显著的脆弱性且鲁棒性表现各异。OI-Bench有望为基于选择界面的LLM指令干扰鲁棒性评估提供更系统化的支持。