Deep Reinforcement Learning (DRL) algorithms have been increasingly employed during the last decade to solve various decision-making problems such as autonomous driving and robotics. However, these algorithms have faced great challenges when deployed in safety-critical environments since they often exhibit erroneous behaviors that can lead to potentially critical errors. One way to assess the safety of DRL agents is to test them to detect possible faults leading to critical failures during their execution. This raises the question of how we can efficiently test DRL policies to ensure their correctness and adherence to safety requirements. Most existing works on testing DRL agents use adversarial attacks that perturb states or actions of the agent. However, such attacks often lead to unrealistic states of the environment. Their main goal is to test the robustness of DRL agents rather than testing the compliance of agents' policies with respect to requirements. Due to the huge state space of DRL environments, the high cost of test execution, and the black-box nature of DRL algorithms, the exhaustive testing of DRL agents is impossible. In this paper, we propose a Search-based Testing Approach of Reinforcement Learning Agents (STARLA) to test the policy of a DRL agent by effectively searching for failing executions of the agent within a limited testing budget. We use machine learning models and a dedicated genetic algorithm to narrow the search towards faulty episodes. We apply STARLA on Deep-Q-Learning agents which are widely used as benchmarks and show that it significantly outperforms Random Testing by detecting more faults related to the agent's policy. We also investigate how to extract rules that characterize faulty episodes of the DRL agent using our search results. Such rules can be used to understand the conditions under which the agent fails and thus assess its deployment risks.

翻译：深度强化学习算法在过去十年中被越来越多地用于解决各类决策问题，例如自动驾驶和机器人技术。然而，这些算法在部署于安全关键环境中时面临巨大挑战，因为它们常常表现出可能导致潜在严重错误的异常行为。评估深度强化学习智能体安全性的一种方法是通过测试来检测其执行过程中可能引发关键性失败的潜在故障。这就引出了一个问题：我们如何高效地测试深度强化学习策略，以确保其正确性并符合安全要求？现有的大多数深度强化学习智能体测试工作使用对抗攻击来扰动智能体的状态或动作。然而，此类攻击往往导致环境状态不切实际。其主要目标是测试深度强化学习智能体的鲁棒性，而非检验智能体策略是否符合需求规范。由于深度强化学习环境的状态空间巨大、测试执行成本高昂以及深度强化学习算法的黑箱特性，对深度强化学习智能体进行全面测试是不可能的。在本文中，我们提出一种面向强化学习智能体的基于搜索的测试方法（STARLA），通过在有限的测试预算内高效搜索智能体的失败执行序列来测试深度强化学习智能体的策略。我们利用机器学习模型和专用的遗传算法将搜索范围缩小至故障片段。我们将STARLA应用于作为广泛使用的基准的深度Q学习智能体，并证明它通过检测更多与智能体策略相关的故障，显著优于随机测试。我们还研究了如何利用我们的搜索结果提取表征深度强化学习智能体故障片段的规则。这些规则可用于理解智能体发生故障的条件，从而评估其部署风险。