Deep Reinforcement Learning (DRL) has achieved remarkable success in domains requiring sequential decision-making, motivating its application to cybersecurity problems. However, transitioning DRL from laboratory simulations to bespoke cyber environments can introduce numerous issues. This is further exacerbated by the often adversarial, non-stationary, and partially-observable nature of most cybersecurity tasks. In this paper, we identify and systematize 11 methodological pitfalls that frequently occur in DRL for cybersecurity (DRL4Sec) literature across the stages of environment modeling, agent training, performance evaluation, and system deployment. By analyzing 66 significant DRL4Sec papers (2018-2025), we quantify the prevalence of each pitfall and find an average of over five pitfalls per paper. We demonstrate the practical impact of these pitfalls using controlled experiments in (i) autonomous cyber defense, (ii) adversarial malware creation, and (iii) web security testing environments. Finally, we provide actionable recommendations for each pitfall to support the development of more rigorous and deployable DRL-based security systems.

翻译：深度强化学习（DRL）在需要序列决策的领域中取得了显著成功，这推动了其在网络安全问题中的应用。然而，将DRL从实验室仿真迁移到定制化的网络环境可能会引入诸多问题。大多数网络安全任务通常具有对抗性、非平稳性和部分可观测性，这进一步加剧了上述挑战。本文识别并系统化了在网络安全领域的深度强化学习（DRL4Sec）文献中，于环境建模、智能体训练、性能评估和系统部署阶段频繁出现的11个方法学陷阱。通过分析66篇重要的DRL4Sec论文（2018-2025年），我们量化了每个陷阱的普遍性，发现平均每篇论文存在超过五个陷阱。我们通过在（i）自主网络防御、（ii）对抗性恶意软件生成以及（iii）Web安全测试环境中的受控实验，展示了这些陷阱的实际影响。最后，我们针对每个陷阱提供了可操作的建议，以支持开发更严谨、更可部署的基于DRL的安全系统。