There is a growing interest in utilizing large-scale language models (LLMs) to advance next-generation Recommender Systems (RecSys), driven by their outstanding language understanding and in-context learning capabilities. In this scenario, tokenizing (i.e., indexing) users and items becomes essential for ensuring a seamless alignment of LLMs with recommendations. While several studies have made progress in representing users and items through textual contents or latent representations, challenges remain in efficiently capturing high-order collaborative knowledge into discrete tokens that are compatible with LLMs. Additionally, the majority of existing tokenization approaches often face difficulties in generalizing effectively to new/unseen users or items that were not in the training corpus. To address these challenges, we propose a novel framework called TokenRec, which introduces not only an effective ID tokenization strategy but also an efficient retrieval paradigm for LLM-based recommendations. Specifically, our tokenization strategy, Masked Vector-Quantized (MQ) Tokenizer, involves quantizing the masked user/item representations learned from collaborative filtering into discrete tokens, thus achieving a smooth incorporation of high-order collaborative knowledge and a generalizable tokenization of users and items for LLM-based RecSys. Meanwhile, our generative retrieval paradigm is designed to efficiently recommend top-$K$ items for users to eliminate the need for the time-consuming auto-regressive decoding and beam search processes used by LLMs, thus significantly reducing inference time. Comprehensive experiments validate the effectiveness of the proposed methods, demonstrating that TokenRec outperforms competitive benchmarks, including both traditional recommender systems and emerging LLM-based recommender systems.

翻译：利用大规模语言模型（LLM）推动下一代推荐系统（RecSys）发展正受到日益广泛的关注，这主要得益于LLM卓越的语言理解能力与上下文学习特性。在此背景下，对用户和项目进行分词（即索引）处理，对于实现LLM与推荐系统的无缝对接至关重要。尽管已有研究通过文本内容或隐式表征在用户与项目表示方面取得进展，但如何将高阶协同知识高效地编码为LLM兼容的离散分词，仍面临诸多挑战。此外，现有分词方法大多难以有效泛化至训练语料中未出现过的新用户或新项目。为应对这些挑战，我们提出名为TokenRec的新型框架，该框架不仅引入了有效的ID分词策略，同时构建了面向LLM推荐的高效检索范式。具体而言，我们的分词策略——掩码向量量化分词器，通过将协同过滤学习得到的掩码化用户/项目表征量化为离散分词，实现了高阶协同知识的平滑注入，并为基于LLM的推荐系统提供了可泛化的用户/项目分词方案。与此同时，我们设计的生成式检索范式能够高效地为用户推荐Top-$K$项目，避免了LLM耗时的自回归解码与束搜索过程，从而显著降低推理耗时。综合实验验证了所提方法的有效性，结果表明TokenRec在包括传统推荐系统与新兴LLM推荐系统在内的多种基准测试中均表现出优越性能。