We implemented and evaluated an automated cyber defense agent. The agent takes security alerts as input and uses reinforcement learning to learn a policy for executing predefined defensive measures. The defender policies were trained in an environment intended to simulate a cyber attack. In the simulation, an attacking agent attempts to capture targets in the environment, while the defender attempts to protect them by enabling defenses. The environment was modeled using attack graphs based on the Meta Attack Language language. We assumed that defensive measures have downtime costs, meaning that the defender agent was penalized for using them. We also assumed that the environment was equipped with an imperfect intrusion detection system that occasionally produces erroneous alerts based on the environment state. To evaluate the setup, we trained the defensive agent with different volumes of intrusion detection system noise. We also trained agents with different attacker strategies and graph sizes. In experiments, the defensive agent using policies trained with reinforcement learning outperformed agents using heuristic policies. Experiments also demonstrated that the policies could generalize across different attacker strategies. However, the performance of the learned policies decreased as the attack graphs increased in size.

翻译：我们实现并评估了一种自动化网络防御智能体。该智能体以安全警报为输入，利用强化学习学习执行预定义防御措施的策略。防御策略在模拟网络攻击的环境中进行训练。在仿真中，攻击智能体试图捕获环境中的目标，而防御智能体则通过启用防御机制来保护这些目标。该环境基于元攻击语言中的攻击图进行建模。我们假设防御措施会产生停机成本，即防御智能体因使用它们而受到惩罚。同时假设环境配备了不完善的入侵检测系统，该系统偶尔会根据环境状态生成错误警报。为评估该设置，我们使用不同噪声水平的入侵检测系统训练防御智能体，并针对不同攻击策略与图规模训练智能体。实验表明，采用强化学习训练策略的防御智能体优于使用启发式策略的智能体。实验还证明，这些策略能够泛化至不同攻击策略。然而，随着攻击图规模增大，学习策略的性能有所下降。