Industrial recommendation systems (RS) rely on the multi-stage pipeline to balance effectiveness and efficiency when delivering items from a vast corpus to users. Existing RS benchmark datasets primarily focus on the exposure space, where novel RS algorithms are trained and evaluated. However, when these algorithms transition to real world industrial RS, they face a critical challenge of handling unexposed items which are a significantly larger space than the exposed one. This discrepancy profoundly impacts their practical performance. Additionally, these algorithms often overlook the intricate interplay between multiple RS stages, resulting in suboptimal overall system performance. To address this issue, we introduce RecFlow, an industrial full flow recommendation dataset designed to bridge the gap between offline RS benchmarks and the real online environment. Unlike existing datasets, RecFlow includes samples not only from the exposure space but also unexposed items filtered at each stage of the RS funnel. Our dataset comprises 38M interactions from 42K users across nearly 9M items with additional 1.9B stage samples collected from 9.3M online requests over 37 days and spanning 6 stages. Leveraging the RecFlow dataset, we conduct courageous exploration experiments, showcasing its potential in designing new algorithms to enhance effectiveness by incorporating stage-specific samples. Some of these algorithms have already been deployed online, consistently yielding significant gains. We propose RecFlow as the first comprehensive benchmark dataset for the RS community, supporting research on designing algorithms at any stage, study of selection bias, debiased algorithms, multi-stage consistency and optimality, multi-task recommendation, and user behavior modeling. The RecFlow dataset, along with the corresponding source code, is available at https://github.com/RecFlow-ICLR/RecFlow.

翻译：工业推荐系统（RS）通常采用多阶段流水线架构，在从海量候选集中向用户呈现物品时平衡推荐效果与系统效率。现有的RS基准数据集主要聚焦于曝光空间，即在此空间内训练和评估新颖的RS算法。然而，当这些算法迁移至真实工业RS环境时，它们面临一个关键挑战：如何处理未曝光物品，该空间规模远大于曝光空间。这种差异会深刻影响算法的实际性能。此外，这些算法往往忽略了多个RS阶段之间复杂的相互作用，导致整体系统性能未能达到最优。为解决这一问题，我们提出了RecFlow——一个旨在弥合离线RS基准与真实在线环境之间差距的工业级全流程推荐数据集。与现有数据集不同，RecFlow不仅包含曝光空间的样本，还涵盖了在RS漏斗各阶段被过滤的未曝光物品。我们的数据集包含来自4.2万用户对近900万物品的3800万次交互，以及基于930万在线请求在37天内跨6个阶段收集的额外19亿条阶段样本。利用RecFlow数据集，我们开展了探索性实验，展示了该数据集在设计新算法以通过融合阶段特定样本来提升推荐效果方面的潜力。其中部分算法已在线部署，并持续带来显著收益。我们提出将RecFlow作为面向RS社区的首个综合性基准数据集，支持以下研究方向：任意阶段的算法设计、选择偏差研究、去偏算法、多阶段一致性与最优性、多任务推荐以及用户行为建模。RecFlow数据集及相应源代码已公开于 https://github.com/RecFlow-ICLR/RecFlow。