Current frameworks for training offensive penetration testing agents with deep reinforcement learning struggle to produce agents that perform well in real-world scenarios, due to the reality gap in simulation-based frameworks and the lack of scalability in emulation-based frameworks. Additionally, existing frameworks often use an unrealistic metric that measures the agents' performance on the training data. NASimEmu, a new framework introduced in this paper, addresses these issues by providing both a simulator and an emulator with a shared interface. This approach allows agents to be trained in simulation and deployed in the emulator, thus verifying the realism of the used abstraction. Our framework promotes the development of general agents that can transfer to novel scenarios unseen during their training. For the simulation part, we adopt an existing simulator NASim and enhance its realism. The emulator is implemented with industry-level tools, such as Vagrant, VirtualBox, and Metasploit. Experiments demonstrate that a simulation-trained agent can be deployed in emulation, and we show how to use the framework to train a general agent that transfers into novel, structurally different scenarios. NASimEmu is available as open-source.

翻译：当前基于深度强化学习的进攻性渗透测试智能体训练框架，由于模拟框架存在现实差距且仿真框架缺乏可扩展性，难以训练出在真实场景中表现良好的智能体。此外，现有框架常采用不切实际的评估指标来衡量智能体在训练数据上的性能。本文提出的新框架NASimEmu通过提供具有共享接口的模拟器与仿真器来解决上述问题。该方法允许智能体在模拟环境中训练后部署至仿真系统，从而验证所用抽象模型的现实性。我们的框架促进开发能够迁移至训练中未见的新场景的通用智能体。在模拟部分，我们采用现有模拟器NASim并增强其现实性；仿真部分则使用行业级工具（如Vagrant、VirtualBox及Metasploit）实现。实验表明，模拟训练的智能体能成功部署至仿真环境，同时我们展示了如何利用该框架训练能迁移至结构不同的新场景的通用智能体。NASimEmu已开放源代码。