Nowadays, the mainstream approach in position allocation system is to utilize a reinforcement learning model to allocate appropriate locations for items in various channels and then mix them into the feed. There are two types of data employed to train reinforcement learning (RL) model for position allocation, named strategy data and random data. Strategy data is collected from the current online model, it suffers from an imbalanced distribution of state-action pairs, resulting in severe overestimation problems during training. On the other hand, random data offers a more uniform distribution of state-action pairs, but is challenging to obtain in industrial scenarios as it could negatively impact platform revenue and user experience due to random exploration. As the two types of data have different distributions, designing an effective strategy to leverage both types of data to enhance the efficacy of the RL model training has become a highly challenging problem. In this study, we propose a framework named Multi-Distribution Data Learning (MDDL) to address the challenge of effectively utilizing both strategy and random data for training RL models on mixed multi-distribution data. Specifically, MDDL incorporates a novel imitation learning signal to mitigate overestimation problems in strategy data and maximizes the RL signal for random data to facilitate effective learning. In our experiments, we evaluated the proposed MDDL framework in a real-world position allocation system and demonstrated its superior performance compared to the previous baseline. MDDL has been fully deployed on the Meituan food delivery platform and currently serves over 300 million users.

翻译：当前，位置分配系统的主流方法是利用强化学习模型为不同通道中的物品分配合适位置，并将其混合至信息流中。用于训练位置分配强化学习模型的数据分为两类：策略数据和随机数据。策略数据来源于当前在线模型，其状态-动作对分布存在不均衡问题，导致训练过程中出现严重的高估偏差。而随机数据虽能提供更均匀的状态-动作对分布，但在工业场景中难以获取——因为随机探索可能损害平台收益与用户体验。由于这两类数据分布不同，如何设计有效策略，同时利用二者提升强化学习模型训练效果已成为极具挑战性的问题。本研究提出名为多分布数据学习（MDDL）的框架，旨在解决混合多分布数据场景下有效利用策略与随机数据训练强化学习模型的难题。具体而言，MDDL通过引入新型模仿学习信号缓解策略数据的高估问题，同时最大化随机数据的强化学习信号以促进有效学习。我们在真实位置分配系统中评估了所提出的MDDL框架，实验结果表明其性能显著优于现有基线方法。目前MDDL已全面部署于美团外卖平台，服务于超过3亿用户。