Network Intrusion Detection Systems (NIDS) are a fundamental tool in cybersecurity. Their ability to generalize across diverse networks is a critical factor in their effectiveness and a prerequisite for real-world applications. In this study, we conduct a comprehensive analysis on the generalization of machine-learning-based NIDS through an extensive experimentation in a cross-dataset framework. We employ four machine learning classifiers and utilize four datasets acquired from different networks: CIC-IDS-2017, CSE-CIC-IDS2018, LycoS-IDS2017, and LycoS-Unicas-IDS2018. Notably, the last dataset is a novel contribution, where we apply corrections based on LycoS-IDS2017 to the well-known CSE-CIC-IDS2018 dataset. The results show nearly perfect classification performance when the models are trained and tested on the same dataset. However, when training and testing the models in a cross-dataset fashion, the classification accuracy is largely commensurate with random chance except for a few combinations of attacks and datasets. We employ data visualization techniques in order to provide valuable insights on the patterns in the data. Our analysis unveils the presence of anomalies in the data that directly hinder the classifiers capability to generalize the learned knowledge to new scenarios. This study enhances our comprehension of the generalization capabilities of machine-learning-based NIDS, highlighting the significance of acknowledging data heterogeneity.

翻译：网络入侵检测系统（NIDS）是网络安全领域的核心工具。其跨异构网络的泛化能力是决定系统有效性的关键因素，也是实现实际应用的必要前提。本研究通过跨数据集框架下的系统性实验，对基于机器学习的NIDS泛化性能进行全面分析。我们采用四种机器学习分类器，并利用从不同网络环境采集的四个数据集：CIC-IDS-2017、CSE-CIC-IDS2018、LycoS-IDS2017和LycoS-Unicas-IDS2018。值得注意的是，最后一个数据集是原创性贡献——我们基于LycoS-IDS2017对著名的CSE-CIC-IDS2018数据集进行了修正。实验结果表明，当模型在同一数据集上训练和测试时，分类性能近乎完美；然而在跨数据集场景下，除少数攻击与数据集的组合外，分类准确率基本等同于随机猜测。我们采用数据可视化技术揭示数据分布的内在模式，分析显示数据中存在的异常直接阻碍了分类器将已学知识泛化至新场景的能力。本研究深化了对基于机器学习的NIDS泛化能力的认知，强调了数据异质性这一关键因素的重要性。