Large language models exhibit surprising sensitivity to the structure of the prompt, but the mechanisms underlying this sensitivity remain poorly understood. In this work, we conduct an in-depth investigation on a striking case: in multiple-choice question answering, placing context before the questions and options (CQO) outperforms the reverse order (QOC) by over 14%p, consistently over a wide range of models and datasets. Through systematic architectural analysis, we identify causal attention as the core mechanism: in QOC prompts, the causal mask prevents option tokens from attending to context, creating an information bottleneck where context becomes invisible to options.

翻译：大型语言模型对提示结构表现出惊人的敏感性，但其背后的机制仍鲜为人知。本研究针对一个显著案例展开深入调查：在多项选择题解答任务中，将语境置于问题与选项之前（CQO）的提示方式，其表现始终优于反向排序（QOC），准确率差距超过14个百分点，这一现象在多种模型与数据集上均保持稳定。通过系统性架构分析，我们确定因果注意力是核心机制：在QOC提示中，因果掩码会阻止选项词元关注语境信息，从而形成语境对选项不可见的信息瓶颈。