While deep learning (DL) models are state-of-the-art in text and image domains, they have not yet consistently outperformed Gradient Boosted Decision Trees (GBDTs) on tabular Learning-To-Rank (LTR) problems. Most of the recent performance gains attained by DL models in text and image tasks have used unsupervised pretraining, which exploits orders of magnitude more unlabeled data than labeled data. To the best of our knowledge, unsupervised pretraining has not been applied to the LTR problem, which often produces vast amounts of unlabeled data. In this work, we study whether unsupervised pretraining of deep models can improve LTR performance over GBDTs and other non-pretrained models. By incorporating simple design choices--including SimCLR-Rank, an LTR-specific pretraining loss--we produce pretrained deep learning models that consistently (across datasets) outperform GBDTs (and other non-pretrained rankers) in the case where there is more unlabeled data than labeled data. This performance improvement occurs not only on average but also on outlier queries. We base our empirical conclusions off of experiments on (1) public benchmark tabular LTR datasets, and (2) a large industry-scale proprietary ranking dataset. Code is provided at https://anonymous.4open.science/r/ltr-pretrain-0DAD/README.md.
翻译:尽管深度学习模型在文本和图像领域已达到最先进水平,但在表格型学习排序(Learning-To-Rank, LTR)问题上,它们尚未持续超越梯度提升决策树(GBDT)。近年来深度学习在文本和图像任务中取得的性能提升,大多依赖于无监督预训练——该方法利用比标记数据多数量级的未标记数据。据我们所知,无监督预训练尚未被应用于LTR问题,而此类问题通常会产生海量未标记数据。本研究探讨深度模型的无监督预训练是否能在LTR任务中超越GBDT及其他非预训练模型。通过引入简单的设计选择(包括面向LTR的预训练损失函数SimCLR-Rank),我们构建的预训练深度学习模型在未标记数据多于标记数据的情况下,能够持续(跨数据集)超越GBDT及其他非预训练排序模型。这种性能提升不仅体现在平均指标上,在异常查询中也同样显著。我们的实验结论基于:(1)公开基准表格型LTR数据集,(2)大型工业级专有排序数据集。代码见https://anonymous.4open.science/r/ltr-pretrain-0DAD/README.md。