Owing to their powerful semantic reasoning capabilities, Large Language Models (LLMs) have been effectively utilized as recommenders, achieving impressive performance. However, the high inference latency of LLMs significantly restricts their practical deployment. To address this issue, this work investigates knowledge distillation from cumbersome LLM-based recommendation models to lightweight conventional sequential models. It encounters three challenges: 1) the teacher's knowledge may not always be reliable; 2) the capacity gap between the teacher and student makes it difficult for the student to assimilate the teacher's knowledge; 3) divergence in semantic space poses a challenge to distill the knowledge from embeddings. To tackle these challenges, this work proposes a novel distillation strategy, DLLM2Rec, specifically tailored for knowledge distillation from LLM-based recommendation models to conventional sequential models. DLLM2Rec comprises: 1) Importance-aware ranking distillation, which filters reliable and student-friendly knowledge by weighting instances according to teacher confidence and student-teacher consistency; 2) Collaborative embedding distillation integrates knowledge from teacher embeddings with collaborative signals mined from the data. Extensive experiments demonstrate the effectiveness of the proposed DLLM2Rec, boosting three typical sequential models with an average improvement of 47.97%, even enabling them to surpass LLM-based recommenders in some cases.

翻译：凭借强大的语义推理能力，大型语言模型（LLMs）已被有效应用于推荐系统并取得显著性能。然而，LLMs的高推理延迟严重制约了其实践部署。为解决该问题，本文研究了从基于LLMs的繁重推荐模型向轻量级传统序列模型进行知识蒸馏的方法。该方法面临三大挑战：1）教师模型知识并非完全可靠；2）师生模型间的能力差距导致学生模型难以吸收教师知识；3）语义空间差异使得从嵌入中蒸馏知识存在困难。针对上述挑战，本文提出新型蒸馏策略DLLM2Rec，专门设计用于将基于LLMs的推荐模型知识迁移至传统序列模型。DLLM2Rec包含：1）重要性感知排序蒸馏，通过基于教师置信度与师生一致性对样本加权，筛选可靠且适合学生模型的知识；2）协作嵌入蒸馏，将教师嵌入知识与从数据中挖掘的协同信号进行整合。大量实验证明DLLM2Rec的有效性，其使三种典型序列模型获得平均47.97%的性能提升，甚至在某些场景下超越基于LLMs的推荐器。