Deep Reinforcement Learning (DRL) algorithms have been increasingly employed during the last decade to solve various decision-making problems such as autonomous driving and robotics. However, these algorithms have faced great challenges when deployed in safety-critical environments since they often exhibit erroneous behaviors that can lead to potentially critical errors. One way to assess the safety of DRL agents is to test them to detect possible faults leading to critical failures during their execution. This raises the question of how we can efficiently test DRL policies to ensure their correctness and adherence to safety requirements. Most existing works on testing DRL agents use adversarial attacks that perturb states or actions of the agent. However, such attacks often lead to unrealistic states of the environment. Their main goal is to test the robustness of DRL agents rather than testing the compliance of agents' policies with respect to requirements. Due to the huge state space of DRL environments, the high cost of test execution, and the black-box nature of DRL algorithms, the exhaustive testing of DRL agents is impossible. In this paper, we propose a Search-based Testing Approach of Reinforcement Learning Agents (STARLA) to test the policy of a DRL agent by effectively searching for failing executions of the agent within a limited testing budget. We use machine learning models and a dedicated genetic algorithm to narrow the search towards faulty episodes. We apply STARLA on Deep-Q-Learning agents which are widely used as benchmarks and show that it significantly outperforms Random Testing by detecting more faults related to the agent's policy. We also investigate how to extract rules that characterize faulty episodes of the DRL agent using our search results. Such rules can be used to understand the conditions under which the agent fails and thus assess its deployment risks.

翻译：深度强化学习（DRL）算法在过去十年中日益广泛地应用于解决各类决策问题，例如自动驾驶和机器人技术。然而，当这些算法部署于安全关键环境时，常因其表现出可能导致潜在严重错误的异常行为而面临巨大挑战。评估DRL智能体安全性的一种途径是通过测试，以检测其执行过程中可能引发关键故障的潜在缺陷。这引出了一个问题：如何高效测试DRL策略以确保其正确性及对安全需求的符合性？现有的大多数DRL智能体测试工作采用对抗攻击方法，通过扰动智能体的状态或动作进行测试。然而，此类攻击往往导致环境状态不切实际，其核心目标是测试DRL智能体的鲁棒性，而非验证智能体策略对需求的符合性。鉴于DRL环境状态空间庞大、测试执行成本高昂以及DRL算法的不透明特性，对DRL智能体进行穷举测试是不可能的。本文提出了一种面向强化学习智能体的搜索测试方法（STARLA），该方法通过在有限测试预算内高效搜索智能体的失败执行路径，来测试DRL智能体的策略。我们利用机器学习模型和专用遗传算法，将搜索范围缩小至故障轨迹。我们将STARLA应用于广泛作为基准的深度Q学习智能体，实验结果表明，该方法在检测智能体策略相关故障方面显著优于随机测试。此外，我们还研究了如何基于搜索结果提取表征DRL智能体故障轨迹的规则，这类规则可用于理解智能体失效的条件，进而评估其部署风险。