In this paper, we implement and evaluate a two-stage retrieval pipeline for a course recommender system that ranks courses for skill-occupation pairs. The in-production recommender system BrightFit provides course recommendations from multiple sources. Some of the course descriptions are long and noisy, while retrieval and ranking in an online system have to be highly efficient. We developed a two-step retrieval pipeline with RankT5 finetuned on MSMARCO as re-ranker. We compare two summarizers for course descriptions: a LongT5 model that we finetuned for the task, and a generative LLM (Vicuna) with in-context learning. We experiment with quantization to reduce the size of the ranking model and increase inference speed. We evaluate our rankers on two newly labelled datasets, with an A/B test, and with a user questionnaire. On the two labelled datasets, our proposed two-stage ranking with automatic summarization achieves a substantial improvement over the in-production (BM25) ranker: nDCG@10 scores improve from 0.482 to 0.684 and from 0.447 to 0.844 on the two datasets. We also achieve a 40% speed-up by using a quantized version of RankT5. The improved quality of the ranking was confirmed by the questionnaire completed by 29 respondents, but not by the A/B test. In the A/B test, a higher clickthrough rate was observed for the BM25-ranking than for the proposed two-stage retrieval. We conclude that T5-based re-ranking and summarization for online course recommendation can obtain much better effectiveness than single-step lexical retrieval, and that quantization has a large effect on RankT5. In the online evaluation, however, other factors than relevance play a role (such as speed and interpretability of the retrieval results), as well as individual preferences.

翻译：本文针对技能-职业配对场景，实现并评估了一种用于课程推荐系统的两阶段检索流水线。实际部署的BrightFit推荐系统需从多源数据中提供课程推荐。部分课程描述冗长且包含噪声，而在线系统的检索与排序必须保持高效。我们开发了采用两阶段检索的流水线，其中使用基于MSMARCO微调的RankT5作为重排序器。针对课程描述摘要任务，我们比较了两种摘要生成器：专门微调的LongT5模型，以及采用上下文学习策略的生成式大语言模型（Vicuna）。通过量化技术减小排序模型规模并提升推理速度。我们在两个新标注数据集上评估排序器性能，同时开展A/B测试与用户问卷调查。在两个标注数据集上，本文提出的自动摘要两阶段排序方法相比现行（BM25）排序器取得显著提升：nDCG@10指标在两个数据集上分别从0.482提升至0.684，从0.447提升至0.844。通过使用量化版RankT5，推理速度提升40%。29位受访者完成的问卷证实了排序质量的改进，但A/B测试结果未呈现相同结论：BM25排序的点击率高于提出的两阶段检索方案。研究结论表明，基于T5的重排序与摘要技术在线课程推荐中能获得远优于单阶段词汇检索的效果，且量化对RankT5性能影响显著。然而在线评估中，除相关性外，检索结果的速度与可解释性以及个体偏好等因素同样发挥作用。