We tackle the Online 3D Bin Packing Problem, a challenging yet practically useful variant of the classical Bin Packing Problem. In this problem, the items are delivered to the agent without informing the full sequence information. Agent must directly pack these items into the target bin stably without changing their arrival order, and no further adjustment is permitted. Online 3D-BPP can be naturally formulated as Markov Decision Process (MDP). We adopt deep reinforcement learning, in particular, the on-policy actor-critic framework, to solve this MDP with constrained action space. To learn a practically feasible packing policy, we propose three critical designs. First, we propose an online analysis of packing stability based on a novel stacking tree. It attains a high analysis accuracy while reducing the computational complexity from $O(N^2)$ to $O(N \log N)$, making it especially suited for RL training. Second, we propose a decoupled packing policy learning for different dimensions of placement which enables high-resolution spatial discretization and hence high packing precision. Third, we introduce a reward function that dictates the robot to place items in a far-to-near order and therefore simplifies the collision avoidance in movement planning of the robotic arm. Furthermore, we provide a comprehensive discussion on several key implemental issues. The extensive evaluation demonstrates that our learned policy outperforms the state-of-the-art methods significantly and is practically usable for real-world applications.

翻译：我们解决在线三维装箱问题，这是经典装箱问题的一个具有挑战性但实际有用的变体。在此问题中，物品在没有告知完整序列信息的情况下被递送给智能体。智能体必须将这些物品稳定地直接装入目标箱子，且不得改变其到达顺序，也不允许进一步调整。在线三维装箱问题可以自然地表述为马尔可夫决策过程。我们采用深度强化学习，特别是同策略演员-评论家框架，来解决这一具有受限动作空间的马尔可夫决策过程。为了学习实际可行的装箱策略，我们提出了三个关键设计。首先，我们提出了一种基于新颖堆叠树的在线稳定性分析。它在将计算复杂度从$O(N^2)$降低到$O(N \log N)$的同时实现了高分析精度，使其特别适用于强化学习训练。其次，我们提出了针对不同放置维度的解耦装箱策略学习，从而实现了高分辨率空间离散化，进而获得高装箱精度。第三，我们引入了一个奖励函数，指示机器人按照从远到近的顺序放置物品，从而简化了机械臂运动规划中的碰撞避免。此外，我们对几个关键实现问题进行了全面讨论。广泛评估表明，我们的学习策略显著优于最先进的方法，并且在实际应用中具有实用价值。