We introduce online action-stacking, an inference-time wrapper for reinforcement learning policies that produces realistic air traffic control commands while allowing training on a much smaller discrete action space. Policies are trained with simple incremental heading or level adjustments, together with an action-damping penalty that reduces instruction frequency and leads agents to issue commands in short bursts. At inference, online action-stacking compiles these bursts of primitive actions into domain-appropriate compound clearances. Using Proximal Policy Optimisation and the BluebirdDT digital twin platform, we train agents to navigate aircraft along lateral routes, manage climb and descent to target flight levels, and perform two-aircraft collision avoidance under a minimum separation constraint. In our lateral navigation experiments, action stacking greatly reduces the number of issued instructions relative to a damped baseline and achieves comparable performance to a policy trained with a 37-dimensional action space, despite operating with only five actions. These results indicate that online action-stacking helps bridge a key gap between standard reinforcement learning formulations and operational ATC requirements, and provides a simple mechanism for scaling to more complex control scenarios.

翻译：我们提出在线动作堆叠，这是一种用于强化学习策略的推理时封装方法，能够生成符合实际的空中交通管制指令，同时允许在更小的离散动作空间上进行训练。策略通过简单的增量航向或高度层调整进行训练，并结合动作阻尼惩罚来降低指令频率，促使智能体以短脉冲形式发出指令。在推理阶段，在线动作堆叠将这些原始动作脉冲编译为符合领域规范的复合许可指令。利用近端策略优化算法和BluebirdDT数字孪生平台，我们训练智能体执行航空器沿横向航路导航、管理爬升与下降至目标飞行高度层，以及在最小间隔约束下实现双机冲突避让。在横向导航实验中，相较于阻尼基线，动作堆叠大幅减少了指令发布数量，并且尽管仅使用五个动作进行操作，其性能仍可与在37维动作空间上训练的策略相媲美。这些结果表明，在线动作堆叠有助于弥合标准强化学习框架与空中交通管制运行需求之间的关键差距，并为扩展至更复杂的控制场景提供了简洁的机制。