Web-scale ranking systems at Meta serving billions of users is complex. Improving ranking models is essential but engineering heavy. Automated Machine Learning (AutoML) can release engineers from labor intensive work of tuning ranking models; however, it is unknown if AutoML is efficient enough to meet tight production timeline in real-world and, at the same time, bring additional improvements to the strong baselines. Moreover, to achieve higher ranking performance, there is an ever-increasing demand to scale up ranking models to even larger capacity, which imposes more challenges on the efficiency. The large scale of models and tight production schedule requires AutoML to outperform human baselines by only using a small number of model evaluation trials (around 100). We presents a sampling-based AutoML method, focusing on neural architecture search and hyperparameter optimization, addressing these challenges in Meta-scale production when building large capacity models. Our approach efficiently handles large-scale data demands. It leverages a lightweight predictor-based searcher and reinforcement learning to explore vast search spaces, significantly reducing the number of model evaluations. Through experiments in large capacity modeling for CTR and CVR applications, we show that our method achieves outstanding Return on Investment (ROI) versus human tuned baselines, with up to 0.09% Normalized Entropy (NE) loss reduction or $25\%$ Query per Second (QPS) increase by only sampling one hundred models on average from a curated search space. The proposed AutoML method has already made real-world impact where a discovered Instagram CTR model with up to -0.36% NE gain (over existing production baseline) was selected for large-scale online A/B test and show statistically significant gain. These production results proved AutoML efficacy and accelerated its adoption in ranking systems at Meta.

翻译：在Meta服务数十亿用户的网络级排序系统复杂性极高。改进排序模型至关重要但工程负担沉重。自动机器学习可解放工程师手动调优排序模型的高强度工作，但其效率能否满足严格的生产周期要求，同时为强基线模型带来额外提升尚不明确。此外，为追求更高排序性能，亟需将排序模型扩展至更大容量，这进一步加剧了效率挑战。模型规模庞大且生产周期紧凑，要求自动机器学习仅通过少量模型评估（约100次）即超越人工基线。本文提出基于采样的自动机器学习方法，聚焦神经架构搜索与超参数优化，以应对Meta级大容量模型构建中的生产挑战。该方法高效处理大规模数据需求，利用轻量级预测器搜索器与强化学习探索广阔搜索空间，大幅减少模型评估次数。通过点击率与转化率场景的大容量建模实验，我们证明该方法相较于人工调优基线实现卓越的投资回报率：在精选搜索空间中仅平均采样100个模型，即可降低高达0.09%的归一化熵或提升25%的每秒查询量。该自动机器学习方法已产生实际影响：其发现的Instagram点击率模型在现有生产基线上实现-0.36%的归一化熵增益，被选入大规模在线A/B测试并取得统计显著性提升。这些生产结果验证了自动机器学习的有效性，并加速其在Meta排序系统中的推广应用。