Most reinforcement learning algorithms are based on a key assumption that Markov decision processes (MDPs) are stationary. However, non-stationary MDPs with dynamic action space are omnipresent in real-world scenarios. Yet problems of dynamic action space reinforcement learning have been studied by many previous works, how to choose valuable actions from new and unseen actions to improve learning efficiency remains unaddressed. To tackle this problem, we propose an intelligent Action Pick-up (AP) algorithm to autonomously choose valuable actions that are most likely to boost performance from a set of new actions. In this paper, we first theoretically analyze and find that a prior optimal policy plays an important role in action pick-up by providing useful knowledge and experience. Then, we design two different AP methods: frequency-based global method and state clustering-based local method, based on the prior optimal policy. Finally, we evaluate the AP on two simulated but challenging environments where action spaces vary over time. Experimental results demonstrate that our proposed AP has advantages over baselines in learning efficiency.

翻译：大多数强化学习算法基于一个关键假设：马尔可夫决策过程是平稳的。然而，现实场景中广泛存在具有动态动作空间的非平稳马尔可夫决策过程。尽管已有诸多工作研究了动态动作空间强化学习问题，但如何从新出现的未知动作中选择有价值动作以提高学习效率仍未得到解决。针对此问题，我们提出了一种智能动作拾取（AP）算法，能够从一组新动作中自主选择最可能提升性能的有价值动作。本文首先通过理论分析发现，先验最优策略通过提供有用知识及经验在动作拾取中发挥重要作用。继而基于先验最优策略设计了两种不同的AP方法：基于频率的全局方法和基于状态聚类的局部方法。最后，在两种动作空间动态变化的仿真挑战环境中评估了AP算法。实验结果表明，所提出的AP方法在学习效率上优于基准方法。