We evaluated 20+ Transformer models for ranking of long documents (including recent LongP models trained with FlashAttention) and compared them with a simple FirstP baseline, which applies the same model to the truncated input (at most 512 tokens). We used MS MARCO Documents v1 as a primary training set and evaluated both the zero-shot transferred and fine-tuned models. On MS MARCO, TREC DLs, and Robust04 no long-document model outperformed FirstP by more than 5% in NDCG and MRR (when averaged over all test sets). We conjectured this was not due to models' inability to process long context, but due to a positional bias of relevant passages, whose distribution was skewed towards the beginning of documents. We found direct evidence of this bias in some test sets, which motivated us to create MS MARCO FarRelevant (based on MS MARCO Passages) where the relevant passages were not present among the first 512 tokens. Unlike standard collections where we saw both little benefit from incorporating longer contexts and limited variability in model performance (within a few %), experiments on MS MARCO FarRelevant uncovered dramatic differences among models. The FirstP models performed roughly at the random-baseline level in both zero-shot and fine-tuning scenarios. Simple aggregation models including MaxP and PARADE Attention had good zero-shot accuracy, but benefited little from fine-tuning. Most other models had poor zero-shot performance (sometimes at a random baseline level), but outstripped MaxP by as much as 13-28% after fine-tuning. Thus, the positional bias not only diminishes benefits of processing longer document contexts, but also leads to model overfitting to positional bias and performing poorly in a zero-shot setting when the distribution of relevant passages changes substantially. We make our software and data available.

翻译：我们评估了20余种用于长文档排序的Transformer模型（包括使用FlashAttention训练的最新LongP模型），并将其与简单的FirstP基线方法进行比较，后者将相同模型应用于截断输入（最多512个词元）。我们以MS MARCO Documents v1作为主要训练集，同时评估了零样本迁移模型与微调模型。在MS MARCO、TREC DLs和Robust04数据集上，所有长文档模型在NDCG和MRR指标上（按所有测试集平均）均未超越FirstP基线超过5%。我们推测这并非源于模型处理长上下文的能力不足，而是由于相关段落存在位置偏差——其分布明显偏向文档开头部分。我们在部分测试集中发现了这种偏差的直接证据，这促使我们基于MS MARCO Passages创建了MS MARCO FarRelevant数据集，其中相关段落均不出现于前512个词元内。在标准数据集中，我们观察到引入更长上下文带来的收益有限，且模型性能差异较小（在几个百分点内）；而在MS MARCO FarRelevant上的实验揭示了模型间的显著差异。FirstP模型在零样本和微调场景下的表现均接近随机基线水平。包括MaxP和PARADE Attention在内的简单聚合模型具有良好的零样本准确率，但微调带来的改进有限。大多数其他模型零样本性能较差（有时处于随机基线水平），但经过微调后以13-28%的优势超越MaxP模型。由此可见，位置偏差不仅削弱了处理更长文档上下文的收益，还会导致模型过度拟合位置偏差，当相关段落分布发生显著变化时，在零样本场景中表现不佳。我们将公开相关软件与数据。