While dense retrieval models have achieved remarkable success, rigorous evaluation of their sensitivity to the position of relevant information (i.e., position bias) remains largely unexplored. Existing benchmarks typically employ position-agnostic relevance labels, conflating the challenge of processing long contexts with the bias against specific evidence locations. To address this challenge, we introduce PosIR (Position-Aware Information Retrieval), a comprehensive benchmark designed to diagnose position bias in diverse retrieval scenarios. PosIR comprises 310 datasets spanning 10 languages and 31 domains, constructed through a rigorous pipeline that ties relevance to precise reference spans, enabling the strict disentanglement of document length from information position. Extensive experiments with 10 state-of-the-art embedding models reveal that: (1) Performance on PosIR in long-context settings correlates poorly with the MMTEB benchmark, exposing limitations in current short-text benchmarks; (2) Position bias is pervasive and intensifies with document length, with most models exhibiting primacy bias while certain models show unexpected recency bias; (3) Gradient-based saliency analysis further uncovers the distinct internal attention mechanisms driving these positional preferences. In summary, PosIR serves as a valuable diagnostic framework to foster the development of position-robust retrieval systems.

翻译：尽管稠密检索模型已取得显著成功，但其对相关信息位置（即位置偏差）敏感性的严格评估在很大程度上仍未得到探索。现有基准通常采用位置无关的相关性标注，将处理长上下文的挑战与针对特定证据位置的偏差混为一谈。为应对这一挑战，我们提出了PosIR（位置感知信息检索），这是一个为诊断多样化检索场景中位置偏差而设计的综合性基准。PosIR包含涵盖10种语言和31个领域的310个数据集，通过将相关性与精确参考片段绑定的严格流程构建而成，从而实现了文档长度与信息位置的严格解耦。使用10个最先进的嵌入模型进行的广泛实验表明：（1）在长上下文设置下，PosIR上的性能与MMTEB基准相关性较弱，揭示了当前短文本基准的局限性；（2）位置偏差普遍存在并随文档长度加剧，大多数模型表现出首因偏差，而某些模型则显示出意外的近因偏差；（3）基于梯度的显著性分析进一步揭示了驱动这些位置偏好的不同内部注意力机制。总之，PosIR作为一个有价值的诊断框架，有助于促进位置鲁棒检索系统的发展。