Understanding user interests is crucial for Click-Through Rate (CTR) prediction tasks. In sequential recommendation, pre-training from user historical behaviors through self-supervised learning can better comprehend user dynamic preferences, presenting the potential for direct integration with CTR tasks. Previous methods have integrated pre-trained models into downstream tasks with the sole purpose of extracting semantic information or well-represented user features, which are then incorporated as new features. However, these approaches tend to ignore the additional inference costs to the downstream tasks, and they do not consider how to transfer the effective information from the pre-trained models for specific estimated items in CTR prediction. In this paper, we propose a Sequential Recommendation Pre-training framework for CTR prediction (SRP4CTR) to tackle the above problems. Initially, we discuss the impact of introducing pre-trained models on inference costs. Subsequently, we introduced a pre-trained method to encode sequence side information concurrently.During the fine-tuning process, we incorporate a cross-attention block to establish a bridge between estimated items and the pre-trained model at a low cost. Moreover, we develop a querying transformer technique to facilitate the knowledge transfer from the pre-trained model to industrial CTR models. Offline and online experiments show that our method outperforms previous baseline models.

翻译：理解用户兴趣对于点击率预测任务至关重要。在序列推荐中，通过自监督学习从用户历史行为进行预训练可以更好地理解用户的动态偏好，这为直接与CTR任务整合提供了潜力。以往的方法将预训练模型集成到下游任务中，其唯一目的是提取语义信息或表征良好的用户特征，然后将这些特征作为新特征融入。然而，这些方法往往忽略了给下游任务带来的额外推理成本，并且没有考虑如何将预训练模型中的有效信息迁移到CTR预测中的特定待估物品上。本文提出了一种用于CTR预测的序列推荐预训练框架，以解决上述问题。首先，我们讨论了引入预训练模型对推理成本的影响。随后，我们引入了一种同时编码序列侧信息的预训练方法。在微调过程中，我们引入了一个交叉注意力模块，以较低成本在待估物品与预训练模型之间建立桥梁。此外，我们开发了一种查询变换器技术，以促进从预训练模型到工业CTR模型的知识迁移。离线和在线实验表明，我们的方法优于以往的基线模型。