With the thriving of pre-trained language model (PLM) widely verified in various of NLP tasks, pioneer efforts attempt to explore the possible cooperation of the general textual information in PLM with the personalized behavioral information in user historical behavior sequences to enhance sequential recommendation (SR). However, despite the commonalities of input format and task goal, there are huge gaps between the behavioral and textual information, which obstruct thoroughly modeling SR as language modeling via PLM. To bridge the gap, we propose a novel Unified pre-trained language model enhanced sequential recommendation (UPSR), aiming to build a unified pre-trained recommendation model for multi-domain recommendation tasks. We formally design five key indicators, namely naturalness, domain consistency, informativeness, noise & ambiguity, and text length, to guide the text-item adaptation and behavior sequence-text sequence adaptation differently for pre-training and fine-tuning stages, which are essential but under-explored by previous works. In experiments, we conduct extensive evaluations on seven datasets with both tuning and zero-shot settings and achieve the overall best performance. Comprehensive model analyses also provide valuable insights for behavior modeling via PLM, shedding light on large pre-trained recommendation models. The source codes will be released in the future.

翻译：随着预训练语言模型（PLM）在各类自然语言处理任务中的广泛验证，先驱性工作尝试探索PLM中的通用文本信息与用户历史行为序列中的个性化行为信息之间的协作，以增强序列推荐（SR）。然而，尽管输入格式与任务目标存在共性，但行为信息与文本信息之间存在巨大鸿沟，这阻碍了通过PLM将序列推荐完全建模为语言建模。为弥合这一鸿沟，我们提出了一种新型的增强序列推荐的统一预训练语言模型（UPSR），旨在构建面向多领域推荐任务的统一预训练推荐模型。我们正式设计了五个关键指标——自然性、领域一致性、信息性、噪声与歧义性以及文本长度，以分别指导预训练与微调阶段的文本-项目适配及行为序列-文本序列适配，这些指标至关重要但此前研究尚未充分探索。在实验中，我们对七个数据集在微调与零样本设置下进行了广泛评估，并取得了整体最佳性能。全面的模型分析还为基于PLM的行为建模提供了宝贵洞见，为大规模预训练推荐模型指明了方向。源代码将在未来公开。