E-commerce search systems such as Taobao Search, the largest e-commerce searching system in China, aim at providing users with the most preferred items (e.g., products). Due to the massive data and limited time for response, a typical industrial ranking system consists of three or more modules, including matching, pre-ranking, and ranking. The pre-ranking is widely considered a mini-ranking module, as it needs to rank hundreds of times more items than the ranking under limited latency. Existing researches focus on building a lighter model that imitates the ranking model. As such, the metric of a pre-ranking model follows the ranking model using Area Under ROC (AUC) for offline evaluation. However, such a metric is inconsistent with online A/B tests in practice, so engineers have to perform costly online tests to reach a convincing conclusion. In our work, we rethink the role of the pre-ranking. We argue that the primary goal of the pre-ranking stage is to return an optimal unordered set rather than an ordered list of items because it is the ranking that determines the final exposures. Since AUC measures the quality of an ordered item list, it is not suitable for evaluating the quality of the output unordered set. This paper proposes a new evaluation metric called All-Scenario Hitrate (ASH) for pre-ranking. ASH is proven effective in the offline evaluation and consistent with online A/B tests based on numerous experiments in Taobao Search. We also introduce an all-scenario-based multi-objective learning framework (ASMOL), which improves the ASH significantly. Surprisingly, the new pre-ranking model can outperforms the ranking model when outputting thousands of items. The phenomenon validates that the pre-ranking stage should not imitate the ranking blindly. With the improvements in ASH consistently translating to online improvement, it makes a 1.2% GMV improvement on Taobao Search.

翻译：电商搜索系统，例如中国最大的电商搜索系统淘宝搜索，旨在为用户提供最偏好的物品（例如商品）。由于海量数据和有限的响应时间，典型的工业级排序系统由三个或更多模块组成，包括匹配、预排序和排序。预排序被广泛视为一个轻量级排序模块，因为在有限的延迟下，它需要对比排序多数百倍的物品进行排序。现有研究集中于构建一个模仿排序模型的更轻量级模型。因此，预排序模型的度量标准沿用排序模型的离线评估指标——ROC曲线下面积（AUC）。然而，这种度量标准在实践中与在线A/B测试结果不一致，导致工程师必须进行成本高昂的在线测试才能得出令人信服的结论。在本工作中，我们重新思考了预排序的作用。我们认为，预排序阶段的主要目标是返回一个最优的无序集合，而非有序物品列表，因为最终曝光顺序由排序阶段决定。由于AUC衡量的是有序物品列表的质量，它不适用于评估输出无序集合的质量。本文提出了一种新的预排序评估指标，称为全场景命中率（ASH）。基于淘宝搜索的大量实验，ASH在离线评估中证明有效，并与在线A/B测试结果一致。我们还引入了一个基于全场景的多目标学习框架（ASMOL），该框架显著提升了ASH。令人惊讶的是，新的预排序模型在输出数千个物品时能够超越排序模型。这一现象验证了预排序阶段不应盲目模仿排序。随着ASH的改进持续转化为在线性能提升，这为淘宝搜索带来了1.2%的GMV增长。