Large language models (LLMs) have shown great promise in recommender systems, where supervised fine-tuning (SFT) is commonly used for adaptation. Subsequent studies further introduce preference learning to incorporate negative samples into the training process. However, existing methods rely on sequence-level, offline-generated negatives, making them less discriminative and informative when adapting LLMs to recommendation tasks with large negative item spaces. To address these challenges, we propose ILRec, a novel preference fine-tuning framework for LLM-based recommendation, leveraging self-hard negative signals extracted from intermediate layers to improve preference learning. Specifically, we identify self-hard negative tokens from intermediate layers as fine-grained negative supervision that dynamically reflects the model's preference learning process. To effectively integrate these signals into training, we design a two-stage framework comprising cross-layer preference optimization and cross-layer preference distillation, enabling the model to jointly discriminate informative negatives and enhance the quality of negative signals from intermediate layers. In addition, we introduce a lightweight collaborative filtering model to assign token-level rewards for negative signals, mitigating the risk of over-penalizing false negatives. Extensive experiments on three datasets demonstrate ILRec's effectiveness in enhancing the performance of LLM-based recommender systems.

翻译：大型语言模型（LLM）在推荐系统中展现出巨大潜力，其中监督微调（SFT）是常用的适应方法。后续研究进一步引入偏好学习，将负样本纳入训练过程。然而，现有方法依赖于序列级、离线生成的负样本，在将LLM适应于具有大规模负物品空间的推荐任务时，其判别性和信息性不足。为解决这些挑战，我们提出ILRec——一种基于LLM推荐的新型偏好微调框架，通过利用从中间层提取的自硬负样本信号来改进偏好学习。具体而言，我们从中间层识别自硬负标记作为细粒度负监督，动态反映模型的偏好学习过程。为有效将这些信号整合到训练中，我们设计了包含跨层偏好优化与跨层偏好蒸馏的两阶段框架，使模型能够联合判别信息性负样本并提升中间层负信号的质量。此外，我们引入轻量级协同过滤模型为负信号分配标记级奖励，以缓解过度惩罚假负样本的风险。在三个数据集上的大量实验证明了ILRec在提升基于LLM的推荐系统性能方面的有效性。