Recent advances in generative artificial intelligence, particularly large language models (LLMs), have opened new opportunities for enhancing recommender systems (RecSys). Most existing LLM-based RecSys approaches operate in a discrete space, using vector-quantized tokenizers to align with the inherent discrete nature of language models. However, these quantization methods often result in lossy tokenization and suboptimal learning, primarily due to inaccurate gradient propagation caused by the non-differentiable argmin operation in standard vector quantization. Inspired by the emerging trend of embracing continuous tokens in language models, we propose ContRec, a novel framework that seamlessly integrates continuous tokens into LLM-based RecSys. Specifically, ContRec consists of two key modules: a sigma-VAE Tokenizer, which encodes users/items with continuous tokens; and a Dispersive Diffusion module, which captures implicit user preference. The tokenizer is trained with a continuous Variational Auto-Encoder (VAE) objective, where three effective techniques are adopted to avoid representation collapse. By conditioning on the previously generated tokens of the LLM backbone during user modeling, the Dispersive Diffusion module performs a conditional diffusion process with a novel Dispersive Loss, enabling high-quality user preference generation through next-token diffusion. Finally, ContRec leverages both the textual reasoning output from the LLM and the latent representations produced by the diffusion model for Top-K item retrieval, thereby delivering comprehensive recommendation results. Extensive experiments on four datasets demonstrate that ContRec consistently outperforms both traditional and SOTA LLM-based recommender systems. Our results highlight the potential of continuous tokenization and generative modeling for advancing the next generation of recommender systems.

翻译：生成式人工智能的最新进展，特别是大语言模型（LLM），为增强推荐系统（RecSys）开辟了新的机遇。现有的大多数基于LLM的推荐系统方法在离散空间中运行，使用向量量化分词器来适应语言模型固有的离散特性。然而，这些量化方法通常会导致有损分词和次优学习，这主要是由于标准向量量化中不可微的argmin操作引起的梯度传播不准确所致。受语言模型中采用连续令牌这一新兴趋势的启发，我们提出了ContRec，这是一个将连续令牌无缝集成到基于LLM的推荐系统中的新颖框架。具体而言，ContRec包含两个关键模块：一个sigma-VAE分词器，用于使用连续令牌对用户/物品进行编码；以及一个扩散分散模块，用于捕获隐式用户偏好。该分词器通过连续变分自编码器（VAE）目标进行训练，其中采用了三种有效技术来避免表示塌缩。通过在用户建模过程中以LLM主干先前生成的令牌为条件，扩散分散模块执行一个带有新颖分散损失的条件扩散过程，从而通过下一令牌扩散实现高质量的用户偏好生成。最后，ContRec同时利用LLM的文本推理输出和扩散模型产生的潜在表示进行Top-K物品检索，从而提供全面的推荐结果。在四个数据集上进行的大量实验表明，ContRec在性能上始终优于传统的和基于LLM的当前最优推荐系统。我们的结果凸显了连续令牌化和生成式建模在推动下一代推荐系统发展方面的潜力。