Machine Learning (ML) has been widely applied to cybersecurity and is considered state-of-the-art for solving many of the open issues in that field. However, it is very difficult to evaluate how good the produced solutions are, since the challenges faced in security may not appear in other areas. One of these challenges is the concept drift, which increases the existing arms race between attackers and defenders: malicious actors can always create novel threats to overcome the defense solutions, which may not consider them in some approaches. Due to this, it is essential to know how to properly build and evaluate an ML-based security solution. In this paper, we identify, detail, and discuss the main challenges in the correct application of ML techniques to cybersecurity data. We evaluate how concept drift, evolution, delayed labels, and adversarial ML impact the existing solutions. Moreover, we address how issues related to data collection affect the quality of the results presented in the security literature, showing that new strategies are needed to improve current solutions. Finally, we present how existing solutions may fail under certain circumstances, and propose mitigations to them, presenting a novel checklist to help the development of future ML solutions for cybersecurity.

翻译：机器学习（ML）已广泛应用于网络安全领域，并被视为解决该领域诸多开放性问题的最先进技术。然而，评估所生成解决方案的质量极为困难，因为安全领域面临的挑战可能与其他领域截然不同。其中一个挑战是概念漂移，它加剧了攻击者与防御者之间业已存在的军备竞赛：恶意行为者总能创造新型威胁以突破防御方案，而某些方法可能未将这些威胁纳入考量。正因如此，掌握如何正确构建并评估基于ML的安全解决方案至关重要。本文识别、详述并探讨了将ML技术正确应用于网络安全数据时面临的主要挑战。我们评估了概念漂移、进化、延迟标签以及对抗性ML对现有解决方案的影响。此外，我们阐释了与数据收集相关的若干问题如何影响安全文献中呈现的结果质量，表明需要新策略来改进当前方案。最后，我们揭示了现有方案在特定情况下可能失效的原因，并提出缓解措施，同时提供了一份新颖的检查清单，以助力未来网络安全ML解决方案的研发。