Owing to their powerful semantic reasoning capabilities, Large Language Models (LLMs) have been effectively utilized as recommenders, achieving impressive performance. However, the high inference latency of LLMs significantly restricts their practical deployment. To address this issue, this work investigates knowledge distillation from cumbersome LLM-based recommendation models to lightweight conventional sequential models. It encounters three challenges: 1) the teacher's knowledge may not always be reliable; 2) the capacity gap between the teacher and student makes it difficult for the student to assimilate the teacher's knowledge; 3) divergence in semantic space poses a challenge to distill the knowledge from embeddings. To tackle these challenges, this work proposes a novel distillation strategy, DLLM2Rec, specifically tailored for knowledge distillation from LLM-based recommendation models to conventional sequential models. DLLM2Rec comprises: 1) Importance-aware ranking distillation, which filters reliable and student-friendly knowledge by weighting instances according to teacher confidence and student-teacher consistency; 2) Collaborative embedding distillation integrates knowledge from teacher embeddings with collaborative signals mined from the data. Extensive experiments demonstrate the effectiveness of the proposed DLLM2Rec, boosting three typical sequential models with an average improvement of 47.97%, even enabling them to surpass LLM-based recommenders in some cases.

翻译：得益于其强大的语义推理能力，大语言模型（LLMs）已被有效地用作推荐器，取得了令人印象深刻的性能。然而，LLMs的高推理延迟极大地限制了其实际部署。为解决此问题，本研究探讨了从笨重的基于LLM的推荐模型到轻量级传统序列模型的知识蒸馏。该过程面临三个挑战：1）教师模型的知识可能并非总是可靠；2）教师与学生模型之间的能力差距使学生难以吸收教师的知识；3）语义空间的差异对从嵌入中蒸馏知识构成挑战。为应对这些挑战，本研究提出了一种新颖的蒸馏策略DLLM2Rec，专门为从基于LLM的推荐模型到传统序列模型的知识蒸馏而设计。DLLM2Rec包括：1）重要性感知排序蒸馏，通过根据教师置信度和师生一致性对实例进行加权，以筛选可靠且对学生友好的知识；2）协作嵌入蒸馏，将教师嵌入的知识与从数据中挖掘的协作信号相结合。大量实验证明了所提出的DLLM2Rec的有效性，它使三种典型序列模型的性能平均提升了47.97%，甚至在某些情况下使其超越了基于LLM的推荐器。