Model evolution and constant availability of data are two common phenomena in large-scale real-world machine learning applications, e.g. ads and recommendation systems. To adapt, the real-world system typically retrain with all available data and online learn with recently available data to update the models periodically with the goal of better serving performance. In this paper, we propose a novel framework, named Confidence Ranking, which designs the optimization objective as a ranking function with two different models. Our confidence ranking loss allows direct optimization of the logits output for different convex surrogate functions of metrics, e.g. AUC and Accuracy depending on the target task and dataset. Armed with our proposed methods, our experiments show that the introduction of confidence ranking loss can outperform all baselines on the CTR prediction tasks of public and industrial datasets. This framework has been deployed in the advertisement system of JD.com to serve the main traffic in the fine-rank stage.

翻译：模型演进与数据持续可用是大规模真实世界机器学习应用（如广告和推荐系统）中的两个常见现象。为适应这些变化，真实系统通常使用所有可用数据重新训练，并利用最新数据进行在线学习，以周期性更新模型来提升服务性能。本文提出一种名为"置信度排序"（Confidence Ranking）的新型框架，该框架将优化目标设计为基于两种不同模型的排序函数。我们的置信度排序损失函数允许针对不同度量指标的凸替代函数（如根据目标任务和数据集选择的AUC和准确率）直接优化logits输出。实验表明，采用所提方法后，引入置信度排序损失能够在公开数据集和工业数据集的点击率预测任务中超越所有基线模型。该框架已部署于京东广告系统的精排阶段，服务于主要流量。