While large transformer models have been successfully used in many real-world applications such as natural language processing, computer vision, and speech processing, scaling transformers for recommender systems remains a challenging problem. Recently, Generative Recommenders framework was proposed to scale beyond typical Deep Learning Recommendation Models (DLRMs). Reformulation of recommendation as sequential transduction task led to improvement of scaling properties in terms of compute. Nevertheless, the largest encoder configuration reported by the HSTU authors amounts only to ~176 million parameters, which is considerably smaller than the hundreds of billions or even trillions of parameters common in modern language models. In this work, we present a recipe for training large transformer recommenders with up to a billion parameters. We show that autoregressive learning on user histories naturally decomposes into two subtasks, feedback prediction and next-item prediction, and demonstrate that such a decomposition scales effectively across a wide range of transformer sizes. Furthermore, we report a successful deployment of our proposed architecture on a large-scale music platform serving millions of users. According to our online A/B tests, this new model increases total listening time by +2.26% and raises the likelihood of user likes by +6.37%, constituting (to our knowledge) the largest improvement in recommendation quality reported for any deep learning-based system in the platform's history.

翻译：尽管大型Transformer模型已成功应用于自然语言处理、计算机视觉和语音处理等诸多实际应用领域，但为推荐系统扩展Transformer模型仍是一个具有挑战性的问题。近期提出的生成式推荐框架旨在突破典型深度学习推荐模型（DLRM）的规模限制。通过将推荐任务重新定义为序列转换问题，该框架在计算效率方面改善了模型的扩展特性。然而，HSTU研究者报告的最大编码器配置仅包含约1.76亿参数，这远小于现代语言模型中常见的数千亿甚至数万亿参数规模。本研究提出了一种可训练高达十亿参数的大型Transformer推荐模型的方法。我们证明，基于用户历史行为的自回归学习可自然分解为反馈预测和下一项目预测两个子任务，并验证了这种分解方式在不同规模的Transformer模型中均能实现有效扩展。此外，我们在服务数百万用户的大规模音乐平台上成功部署了所提出的架构。根据在线A/B测试结果，新模型使总收听时长提升+2.26%，用户点赞概率提高+6.37%，据我们所知，这是该平台历史上基于深度学习的推荐系统所实现的最大质量改进。