Recently, deep learning-based models have been widely studied for click-through rate (CTR) prediction and lead to improved prediction accuracy in many industrial applications. However, current research focuses primarily on building complex network architectures to better capture sophisticated feature interactions and dynamic user behaviors. The increased model complexity may slow down online inference and hinder its adoption in real-time applications. Instead, our work targets at a new model training strategy based on knowledge distillation (KD). KD is a teacher-student learning framework to transfer knowledge learned from a teacher model to a student model. The KD strategy not only allows us to simplify the student model as a vanilla DNN model but also achieves significant accuracy improvements over the state-of-the-art teacher models. The benefits thus motivate us to further explore the use of a powerful ensemble of teachers for more accurate student model training. We also propose some novel techniques to facilitate ensembled CTR prediction, including teacher gating and early stopping by distillation loss. We conduct comprehensive experiments against 12 existing models and across three industrial datasets. Both offline and online A/B testing results show the effectiveness of our KD-based training strategy.

翻译：近期，基于深度学习的模型在点击率（CTR）预测领域得到广泛研究，并在众多工业应用中显著提升了预测精度。然而，当前研究主要聚焦于构建复杂的网络架构，以更有效地捕捉特征间的复杂交互和用户动态行为。模型复杂度的增加可能导致在线推理速度下降，从而限制其在实时场景中的应用。为此，本文提出一种基于知识蒸馏（KD）的新型模型训练策略。知识蒸馏是一种师生学习框架，旨在将教师模型习得的知识迁移至学生模型。该策略不仅允许将学生模型简化为标准的DNN模型，还能在预测精度上显著超越现有最优教师模型。这一优势进一步促使我们探索利用强集成教师模型来训练更精准的学生模型。此外，我们提出若干创新技术以优化集成CTR预测，包括教师门控机制和基于蒸馏损失的早停法。我们在三个工业数据集上，针对12个现有模型开展了全面实验。离线与在线A/B测试结果均验证了基于知识蒸馏的训练策略的有效性。