Unbiased learning-to-rank (ULTR) is a well-established framework for learning from user clicks, which are often biased by the ranker collecting the data. While theoretically justified and extensively tested in simulation, ULTR techniques lack empirical validation, especially on modern search engines. The dataset released for the WSDM Cup 2023, collected from Baidu's search engine, offers a rare opportunity to assess the real-world performance of prominent ULTR techniques. Despite multiple submissions during the WSDM Cup 2023 and the subsequent NTCIR ULTRE-2 task, it remains unclear whether the observed improvements stem from applying ULTR or other learning techniques. We revisit and extend the available experiments. We find that unbiased learning-to-rank techniques do not bring clear performance improvements, especially compared to the stark differences brought by the choice of ranking loss and query-document features. Our experiments reveal that ULTR robustly improves click prediction. However, these gains in click prediction do not translate to enhanced ranking performance on expert relevance annotations, implying that conclusions strongly depend on how success is measured in this benchmark.

翻译：无偏学习排序（ULTR）是一种从用户点击中学习的成熟框架，而用户点击往往会受到收集数据的排序器产生的偏差影响。尽管在理论上经过验证并在模拟中广泛测试，ULTR技术仍缺乏实证验证，特别是在现代搜索引擎上的应用。WSDM Cup 2023发布的来自百度搜索引擎的数据集，为评估主流ULTR技术的真实世界性能提供了罕见的机会。尽管在WSDM Cup 2023及后续的NTCIR ULTRE-2任务中已有多次提交，但仍不清楚观察到的改进是源于应用ULTR还是其他学习技术。我们重新审视并扩展了现有实验。我们发现，无偏学习排序技术并未带来明显的性能提升，尤其是与选择排序损失和查询-文档特征所带来的显著差异相比。我们的实验表明，ULTR能稳健地改善点击预测。然而，这些点击预测的提升并未转化为专家相关性标注上的增强排序性能，这意味着结论强烈依赖于在该基准测试中如何衡量成功。