Modern recommender systems usually present items as a streaming, one-dimensional ranking list. Recently there is a trend in e-commerce that the recommended items are organized grid-based panels with two dimensions where users can view the items in both vertical and horizontal directions. Presenting items in grid-based result panels poses new challenges to recommender systems because existing models are all designed to output sequential lists while the slots in a grid-based panel have no explicit order. Directly converting the item rankings into grids (e.g., pre-defining an order on the slots) overlooks the user-specific behavioral patterns on grid-based panels and inevitably hurts the user experiences. To address this issue, we propose a novel Markov decision process (MDP) to place the items in 2D grid-based result panels at the final re-ranking stage of the recommender systems. The model, referred to as Panel-MDP, takes an initial item ranking from the early stages as the input. Then, it defines \emph{the MDP discrete time steps as the ranks in the initial ranking list, and the actions as the prediction of the user-item preference and the selection of the slots}. At each time step, Panel-MDP sequentially executes two sub-actions: first deciding whether the current item in the initial ranking list is preferred by the user; then selecting a slot for placing the item if preferred, or skipping the item otherwise. The process is continued until all of the panel slots are filled. The reinforcement learning algorithm of PPO is employed to implement and learn the parameters in the Panel-MDP. Simulation and experiments on a dataset collected from a widely-used e-commerce app demonstrated the superiority of Panel-MDP in terms of recommending 2D grid-based result panels.

翻译：现代推荐系统通常以流式一维排序列表的形式展示推荐内容。近期电子商务领域出现新趋势，推荐项目被组织为具有二维网格结构的展示面板，用户可在垂直与水平两个方向浏览内容。网格化结果面板的呈现方式给推荐系统带来新挑战：现有模型均设计为输出序列化列表，而网格面板中的展示槽位并无明确顺序。直接将项目排序转换为网格（如预定义槽位顺序）会忽略用户在网格面板上的行为模式，不可避免地损害用户体验。为解决该问题，我们提出一种新颖的马尔可夫决策过程，在推荐系统的最终重排序阶段将项目放置于二维网格结果面板中。该模型名为Panel-MDP，以早期阶段生成的初始排序列表为输入，将马尔可夫决策过程的离散时间步定义为初始排序列表中的排名，将动作定义为用户-项目偏好预测与槽位选择。在每个时间步，Panel-MDP顺序执行两个子动作：首先判定初始排序列表中的当前项目是否被用户偏好；若被偏好则选择槽位放置该项目，否则跳过该项目。该过程持续直至面板所有槽位填满。我们采用PPO强化学习算法实现Panel-MDP并学习其参数。基于某广泛使用的电商应用收集数据进行的仿真与实验表明，Panel-MDP在推荐二维网格结果面板方面具有显著优越性。