Neural Architecture Search (NAS) has demonstrated its efficacy in computer vision and potential for ranking systems. However, prior work focused on academic problems, which are evaluated at small scale under well-controlled fixed baselines. In industry system, such as ranking system in Meta, it is unclear whether NAS algorithms from the literature can outperform production baselines because of: (1) scale - Meta ranking systems serve billions of users, (2) strong baselines - the baselines are production models optimized by hundreds to thousands of world-class engineers for years since the rise of deep learning, (3) dynamic baselines - engineers may have established new and stronger baselines during NAS search, and (4) efficiency - the search pipeline must yield results quickly in alignment with the productionization life cycle. In this paper, we present Rankitect, a NAS software framework for ranking systems at Meta. Rankitect seeks to build brand new architectures by composing low level building blocks from scratch. Rankitect implements and improves state-of-the-art (SOTA) NAS methods for comprehensive and fair comparison under the same search space, including sampling-based NAS, one-shot NAS, and Differentiable NAS (DNAS). We evaluate Rankitect by comparing to multiple production ranking models at Meta. We find that Rankitect can discover new models from scratch achieving competitive tradeoff between Normalized Entropy loss and FLOPs. When utilizing search space designed by engineers, Rankitect can generate better models than engineers, achieving positive offline evaluation and online A/B test at Meta scale.

翻译：摘要：神经架构搜索（NAS）在计算机视觉领域已展现出其有效性，并在排序系统中显示出潜力。然而，先前的研究聚焦于学术问题，这些研究在严格控制的固定基线下基于小规模数据进行评估。在工业系统中，例如Meta的排序系统，现有文献中的NAS算法是否能够超越生产基线尚不明确，原因在于：（1）规模——Meta排序系统服务数十亿用户；（2）强基线——这些基线是基于深度学习兴起以来由数百至数千名世界级工程师历经多年优化的生产模型；（3）动态基线——在NAS搜索过程中，工程师可能已建立新的、更强的基线；（4）效率——搜索流程必须与产品化生命周期对齐，快速产出结果。本文提出Rankitect，这是一个面向Meta排序系统的NAS软件框架。Rankitect旨在通过从零开始组合底层构建模块，构建全新的架构。Rankitect实现并改进了最先进的NAS方法（包括基于采样的NAS、单次NAS和可微分NAS）以在同一搜索空间内进行公平且全面的比较。我们通过将Rankitect与Meta的多个生产排序模型进行对比来评估其性能。研究发现，Rankitect能从零开始发现新模型，在归一化熵损失与FLOPs之间实现具有竞争力的权衡。当利用工程师设计的搜索空间时，Rankitect能生成优于工程师的模型，并在Meta规模下取得积极的离线评估和在线A/B测试结果。