Large language models (LLMs) have gained much attention in the recommendation community; some studies have observed that LLMs, fine-tuned by the cross-entropy loss with a full softmax, could achieve state-of-the-art performance already. However, these claims are drawn from unobjective and unfair comparisons. In view of the substantial quantity of items in reality, conventional recommenders typically adopt a pointwise/pairwise loss function instead for training. This substitute however causes severe performance degradation, leading to under-estimation of conventional methods and over-confidence in the ranking capability of LLMs. In this work, we theoretically justify the superiority of cross-entropy, and showcase that it can be adequately replaced by some elementary approximations with certain necessary modifications. The remarkable results across three public datasets corroborate that even in a practical sense, existing LLM-based methods are not as effective as claimed for next-item recommendation. We hope that these theoretical understandings in conjunction with the empirical results will facilitate an objective evaluation of LLM-based recommendation in the future.

翻译：大语言模型在推荐领域引起了广泛关注；一些研究发现，通过全softmax交叉熵损失微调的大语言模型已经能够达到最先进的性能。然而，这些结论基于不客观和不公平的比较。考虑到现实中物品数量庞大，传统推荐方法通常采用逐点/成对损失函数进行训练。但这种替代方法会导致严重的性能退化，从而低估了传统方法的效果，并高估了大语言模型的排序能力。本研究从理论上论证了交叉熵的优越性，并展示了通过某些必要的修改，可以用一些基本近似方法充分替代它。在三个公开数据集上的显著结果证实，即使在实践意义上，现有基于大语言模型的方法在下一次物品推荐方面并未如宣称的那样有效。我们希望这些理论理解和实证结果能够促进未来对基于大语言模型推荐系统的客观评估。