Although BERT-based ranking models have been commonly used in commercial search engines, they are usually time-consuming for online ranking tasks. Knowledge distillation, which aims at learning a smaller model with comparable performance to a larger model, is a common strategy for reducing the online inference latency. In this paper, we investigate the effect of different loss functions for uniform-architecture distillation of BERT-based ranking models. Here "uniform-architecture" denotes that both teacher and student models are in cross-encoder architecture, while the student models include small-scaled pre-trained language models. Our experimental results reveal that the optimal distillation configuration for ranking tasks is much different than general natural language processing tasks. Specifically, when the student models are in cross-encoder architecture, a pairwise loss of hard labels is critical for training student models, whereas the distillation objectives of intermediate Transformer layers may hurt performance. These findings emphasize the necessity of carefully designing a distillation strategy (for cross-encoder student models) tailored for document ranking with pairwise training samples.

翻译：尽管基于BERT的排序模型已在商业搜索引擎中广泛应用，但其在线排序任务通常耗时较长。知识蒸馏——旨在学习一个性能与更大模型相当的小型模型——是降低在线推理延迟的常用策略。本文研究了不同损失函数对基于BERT的排序模型统一架构蒸馏的影响。这里的"统一架构"指教师模型与学生模型均采用交叉编码器架构，而学生模型包含小规模预训练语言模型。实验结果表明，排序任务的最优蒸馏配置与通用自然语言处理任务存在显著差异。具体而言，当学生模型采用交叉编码器架构时，硬标签的成对损失对训练学生模型至关重要，而中间Transformer层的蒸馏目标可能会损害性能。这些发现强调了针对基于成对训练样本的文档排序任务，精心设计（面向交叉编码器学生模型的）蒸馏策略的必要性。