Recent advances in Information Retrieval have established transformer-based cross-encoders as a keystone in IR. Recent studies have focused on knowledge distillation and showed that, with the right strategy, traditional cross-encoders could reach the level of effectiveness of LLM re-rankers. Yet, comparisons with previous training strategies, including distillation from strong cross-encoder teachers, remain unclear. In addition, few studies cover a similar range of backbone encoders, while substantial improvements have been made in this area since BERT. This lack of comprehensive studies in controlled environments makes it difficult to identify robust design choices. In this work, we reproduce \citet{schlattRankDistiLLMClosingEffectiveness2025} LLM-based distillation strategy and compare it to \citet{hofstatterImprovingEfficientNeural2020} approach based on an ensemble of cross-encoder teachers, as well as other supervised objectives, to fine-tune a large range of cross-encoders, from the original BERT and its follow-ups RoBERTa, ELECTRA and DeBERTa-v3, to the more recent ModernBERT. We evaluate all models on both in-domain (TREC-DL and MS~MARCO dev) and out-of-domain datasets (BEIR, LoTTE, and Robust04). Our results show that objectives emphasizing relative comparisons -- pairwise MarginMSE and listwise InfoNCE -- consistently outperform pointwise baselines across all backbones and evaluation settings, and that objective choice can yield gains comparable to scaling the backbone architecture.

翻译：信息检索领域的最新进展确立了基于Transformer的交叉编码器在IR中的关键地位。近期研究聚焦于知识蒸馏，并表明采用恰当策略时，传统交叉编码器能达到LLM重排序器的效能水平。然而，与先前训练策略（包括从强交叉编码器教师模型进行蒸馏）的比较仍不明确。此外，鲜有研究覆盖类似范围的骨干编码器，而自BERT以来该领域已取得显著改进。受控环境中综合性研究的缺乏，使得难以确定稳健的设计选择。本工作中，我们复现了\citet{schlattRankDistiLLMClosingEffectiveness2025}基于LLM的蒸馏策略，并将其与\citet{hofstatterImprovingEfficientNeural2020}基于交叉编码器教师模型集成的方法以及其他监督目标进行比较，以微调从原始BERT及其后续模型RoBERTa、ELECTRA和DeBERTa-v3到更近期的ModernBERT在内的大量交叉编码器。我们在领域内（TREC-DL和MS~MARCO开发集）与领域外数据集（BEIR、LoTTE和Robust04）上评估所有模型。结果表明：强调相对比较的目标函数——成对MarginMSE与列表式InfoNCE——在所有骨干架构和评估设置中均持续优于逐点基线，且目标函数选择带来的性能增益可与扩展骨干架构相媲美。