With the thriving of pre-trained language model (PLM) widely verified in various of NLP tasks, pioneer efforts attempt to explore the possible cooperation of the general textual information in PLM with the personalized behavioral information in user historical behavior sequences to enhance sequential recommendation (SR). However, despite the commonalities of input format and task goal, there are huge gaps between the behavioral and textual information, which obstruct thoroughly modeling SR as language modeling via PLM. To bridge the gap, we propose a novel Unified pre-trained language model enhanced sequential recommendation (UPSR), aiming to build a unified pre-trained recommendation model for multi-domain recommendation tasks. We formally design five key indicators, namely naturalness, domain consistency, informativeness, noise & ambiguity, and text length, to guide the text->item adaptation and behavior sequence->text sequence adaptation differently for pre-training and fine-tuning stages, which are essential but under-explored by previous works. In experiments, we conduct extensive evaluations on seven datasets with both tuning and zero-shot settings and achieve the overall best performance. Comprehensive model analyses also provide valuable insights for behavior modeling via PLM, shedding light on large pre-trained recommendation models. The source codes will be released in the future.

翻译：随着预训练语言模型（PLM）在各类自然语言处理任务中的广泛验证，先驱研究尝试探索PLM中的通用文本信息与用户历史行为序列中的个性化行为信息之间的协同作用，以增强序列推荐（SR）。然而，尽管输入格式和任务目标存在共性，行为信息与文本信息之间仍存在巨大鸿沟，这阻碍了通过PLM将序列推荐完全建模为语言模型。为弥合这一差距，我们提出了一种新颖的统一预训练语言模型增强的序列推荐方法（UPSR），旨在构建面向多领域推荐任务的统一预训练推荐模型。我们正式设计了五个关键指标，即自然性、领域一致性、信息量、噪声与歧义性以及文本长度，以指导预训练和微调阶段中文本到项目的适配以及行为序列到文本序列的适配。这些指标至关重要但此前研究尚未充分探索。在实验中，我们针对七个数据集在微调和零样本设置下进行了广泛评估，并取得了整体最优性能。全面的模型分析还为通过PLM进行行为建模提供了宝贵见解，为大规模预训练推荐模型指明了方向。源代码将在未来发布。