The Markov decision process (MDP) provides a mathematical framework for modeling sequential decision-making problems, many of which are crucial to security and safety, such as autonomous driving and robot control. The rapid development of artificial intelligence research has created efficient methods for solving MDPs, such as deep neural networks (DNNs), reinforcement learning (RL), and imitation learning (IL). However, these popular models solving MDPs are neither thoroughly tested nor rigorously reliable. We present MDPFuzz, the first blackbox fuzz testing framework for models solving MDPs. MDPFuzz forms testing oracles by checking whether the target model enters abnormal and dangerous states. During fuzzing, MDPFuzz decides which mutated state to retain by measuring if it can reduce cumulative rewards or form a new state sequence. We design efficient techniques to quantify the "freshness" of a state sequence using Gaussian mixture models (GMMs) and dynamic expectation-maximization (DynEM). We also prioritize states with high potential of revealing crashes by estimating the local sensitivity of target models over states. MDPFuzz is evaluated on five state-of-the-art models for solving MDPs, including supervised DNN, RL, IL, and multi-agent RL. Our evaluation includes scenarios of autonomous driving, aircraft collision avoidance, and two games that are often used to benchmark RL. During a 12-hour run, we find over 80 crash-triggering state sequences on each model. We show inspiring findings that crash-triggering states, though they look normal, induce distinct neuron activation patterns compared with normal states. We further develop an abnormal behavior detector to harden all the evaluated models and repair them with the findings of MDPFuzz to significantly enhance their robustness without sacrificing accuracy.

翻译：马尔可夫决策过程（MDP）为建模序贯决策问题提供了数学框架，其中许多问题对安全性和可靠性至关重要，例如自动驾驶和机器人控制。人工智能研究的快速发展催生了求解MDP的高效方法，如深度神经网络（DNN）、强化学习（RL）和模仿学习（IL）。然而，这些求解MDP的流行模型既未经过充分测试，也缺乏严格的可靠性保障。我们提出MDPFuzz——首个针对求解MDP模型的黑盒模糊测试框架。MDPFuzz通过检查目标模型是否进入异常和危险状态来构建测试预言。在模糊测试过程中，MDPFuzz通过判断变异状态能否降低累积奖励或形成新状态序列来决定保留哪些变异状态。我们设计了利用高斯混合模型（GMM）和动态期望最大化（DynEM）量化状态序列"新鲜度"的高效技术，并通过估计目标模型在状态上的局部敏感度，优先处理具有高崩溃发现潜力的状态。我们在五种求解MDP的最先进模型上评估了MDPFuzz，包括监督式DNN、强化学习、模仿学习及多智能体强化学习。评估场景涵盖自动驾驶、飞机防撞及两个常用于强化学习基准测试的游戏。在12小时的运行中，每个模型均被发现超过80个触发崩溃的状态序列。我们揭示了启发性发现：触发崩溃的状态虽然看似正常，但与正常状态相比会诱发不同的神经元激活模式。我们进一步开发了异常行为检测器来强化所有被评估模型，并利用MDPFuzz的发现对其进行修复，在保证准确率不降低的前提下显著提升鲁棒性。