With the thriving of pre-trained language model (PLM) widely verified in various of NLP tasks, pioneer efforts attempt to explore the possible cooperation of the general textual information in PLM with the personalized behavioral information in user historical behavior sequences to enhance sequential recommendation (SR). However, despite the commonalities of input format and task goal, there are huge gaps between the behavioral and textual information, which obstruct thoroughly modeling SR as language modeling via PLM. To bridge the gap, we propose a novel Unified pre-trained language model enhanced sequential recommendation (UPSR), aiming to build a unified pre-trained recommendation model for multi-domain recommendation tasks. We formally design five key indicators, namely naturalness, domain consistency, informativeness, noise & ambiguity, and text length, to guide the text->item adaptation and behavior sequence->text sequence adaptation differently for pre-training and fine-tuning stages, which are essential but under-explored by previous works. In experiments, we conduct extensive evaluations on seven datasets with both tuning and zero-shot settings and achieve the overall best performance. Comprehensive model analyses also provide valuable insights for behavior modeling via PLM, shedding light on large pre-trained recommendation models. The source codes will be released in the future.

翻译：随着预训练语言模型（PLM）在各类自然语言处理任务中的广泛验证，先驱性尝试探索将PLM中的通用文本信息与用户历史行为序列中的个性化行为信息协同结合，以增强序列推荐（SR）。然而，尽管输入格式与任务目标具有共性，行为信息与文本信息之间仍存在巨大鸿沟，这阻碍了通过PLM将序列推荐彻底建模为语言建模。为弥合这一鸿沟，我们提出一种新型统一预训练语言模型增强序列推荐（UPSR），旨在针对多领域推荐任务构建统一的预训练推荐模型。我们正式设计了五项关键指标，即自然性、领域一致性、信息量、噪声与歧义性及文本长度，以分别在预训练与微调阶段引导文本到物品的适配以及行为序列到文本序列的适配——这些指标至关重要但此前研究未予充分探索。实验中，我们在七个数据集上进行了调优与零样本设置的全面评估，取得了整体最优性能。全面的模型分析也为通过PLM进行行为建模提供了宝贵见解，为大规模预训练推荐模型研究指明了方向。源代码将于未来公开。