Retrieval benchmarks for large language models (LLMs) should reflect the long, reasoning-intensive queries typical of retrieval-augmented generation (RAG). We present a systematic study of BRIGHT, a reasoning-focused retrieval benchmark, along with strong, reproducible reference methods integrated into Anserini, Pyserini, and RankLLM. We evaluate lexical, sparse, dense, and fusion-based retrievers, as well as LLM rerankers, under long-query settings. In reproducing BRIGHT's lexical baseline, we identify a key under-documented detail: query-side BM25 (BM25Q), which applies BM25 weighting to the query itself. On long, multi-sentence queries, BM25Q consistently outperforms standard BM25, making it the strongest lexical baseline for reasoning-oriented retrieval. We further audit the BRIGHT corpus, uncovering data quality issues that impact evaluation, and offer mitigation. Finally, we study the generalizability of BM25Q across five additional benchmarks, finding its gains largely specific to BRIGHT, while fusion with standard BM25 provides the most consistent improvements across datasets.

翻译：面向大语言模型（LLM）的检索基准应反映检索增强生成（RAG）中常见的冗长、推理密集型查询。我们对BRIGHT——这一聚焦推理的检索基准——进行了系统性研究，并提供了集成于Anserini、Pyserini和RankLLM中强健且可复现的参考方法。我们在长查询场景下评估了词法、稀疏、稠密及融合检索器，以及LLM重排序器。在复现BRIGHT词法基线的过程中，我们发现了一个关键但未充分记录的细节：查询端BM25（BM25Q），即对查询本身应用BM25加权。针对长句多段落查询，BM25Q始终优于标准BM25，成为推理导向检索中最强的词法基线。我们进一步审校了BRIGHT语料库，发现了影响评估的数据质量问题，并提出了缓解方案。最后，我们研究了BM25Q在五个额外基准上的泛化能力，发现其优势主要局限于BRIGHT，而标准BM25与BM25Q的融合方法在跨数据集上持续带来最显著的改进。