Deep reinforcement learning (DRL) has demonstrated its potential in solving complex manufacturing decision-making problems, especially in a context where the system learns over time with actual operation in the absence of training data. One interesting and challenging application for such methods is the assembly sequence planning (ASP) problem. In this paper, we propose an approach to the implementation of DRL methods in ASP. The proposed approach introduces in the RL environment parametric actions to improve training time and sample efficiency and uses two different reward signals: (1) user's preferences and (2) total assembly time duration. The user's preferences signal addresses the difficulties and non-ergonomic properties of the assembly faced by the human and the total assembly time signal enforces the optimization of the assembly. Three of the most powerful deep RL methods were studied, Advantage Actor-Critic (A2C), Deep Q-Learning (DQN), and Rainbow, in two different scenarios: a stochastic and a deterministic one. Finally, the performance of the DRL algorithms was compared to tabular Q-Learnings performance. After 10,000 episodes, the system achieved near optimal behaviour for the algorithms tabular Q-Learning, A2C, and Rainbow. Though, for more complex scenarios, the algorithm tabular Q-Learning is expected to underperform in comparison to the other 2 algorithms. The results support the potential for the application of deep reinforcement learning in assembly sequence planning problems with human interaction.

翻译：深度强化学习（DRL）在解决复杂制造决策问题中展现出巨大潜力，尤其适用于系统在缺乏训练数据的情况下通过实际运行逐步学习的场景。装配序列规划（ASP）问题是此类方法的一个具有挑战性的重要应用方向。本文提出了一种将DRL方法应用于ASP问题的实现方案。该方案在强化学习环境中引入参数化动作以提升训练效率与样本利用率，并采用两种不同的奖励信号：（1）用户偏好和（2）总装配时间。用户偏好信号用于应对人工装配中的困难与非人体工学特性，而总装配时间信号则推动装配过程的优化。本研究在随机与确定性两种场景下，对三种最先进的深度强化学习方法——优势演员-评论家（A2C）、深度Q学习（DQN）和Rainbow——进行了分析。最后，将DRL算法的性能与传统表格Q学习算法进行对比。经过10,000轮训练后，表格Q学习、A2C和Rainbow算法均实现了接近最优的行为。然而，在更复杂的场景下，表格Q学习算法的性能预计将劣于其他两种算法。研究结果证实了深度强化学习在包含人机交互的装配序列规划问题中的应用潜力。