News recommendation aims to predict click behaviors based on user behaviors. How to effectively model the user representations is the key to recommending preferred news. Existing works are mostly focused on improvements in the supervised fine-tuning stage. However, there is still a lack of PLM-based unsupervised pre-training methods optimized for user representations. In this work, we propose an unsupervised pre-training paradigm with two tasks, i.e. user behavior masking and user behavior generation, both towards effective user behavior modeling. Firstly, we introduce the user behavior masking pre-training task to recover the masked user behaviors based on their contextual behaviors. In this way, the model could capture a much stronger and more comprehensive user news reading pattern. Besides, we incorporate a novel auxiliary user behavior generation pre-training task to enhance the user representation vector derived from the user encoder. We use the above pre-trained user modeling encoder to obtain news and user representations in downstream fine-tuning. Evaluations on the real-world news benchmark show significant performance improvements over existing baselines.

翻译：新闻推荐旨在根据用户行为预测点击行为，而如何有效建模用户表示是推荐偏好新闻的关键。现有工作主要集中于监督微调阶段的改进，但针对用户表示的基于预训练语言模型的无监督预训练方法仍显不足。本文提出一种包含两项任务的无监督预训练范式，即用户行为遮蔽与用户行为生成，两者均致力于实现高效的用户行为建模。首先，我们引入用户行为遮蔽预训练任务，通过上下文行为恢复被遮蔽的用户行为，从而使模型能够捕获更强且更全面的用户新闻阅读模式。此外，我们创新性地引入辅助的用户行为生成预训练任务，以增强用户编码器输出的用户表示向量。在下游微调中，使用上述预训练的用户建模编码器获取新闻表示和用户表示。在真实新闻基准上的评估表明，该方法相较于现有基线取得了显著的性能提升。