The performance of Machine Learning (ML) and Deep Learning (DL)-based Intrusion Detection and Prevention Systems (IDS/IPS) is critically dependent on the relevance and quality of the datasets used for training and evaluation. However, current AI model evaluation practices for developing IDS/IPS focus predominantly on accuracy metrics, often overlooking whether datasets represent industry-specific threats. To address this gap, we introduce a novel multi-dimensional framework that integrates the MITRE ATT&CK knowledge base for threat intelligence and employs five complementary metrics that together provide a comprehensive assessment of dataset suitability. Methodologically, this framework combines threat intelligence, natural language processing, and quantitative analysis to assess the suitability of datasets for specific industry contexts. Applying this framework to nine publicly available IDS/IPS datasets reveals significant gaps in threat coverage, particularly in the healthcare, energy, and financial sectors. In particular, recent datasets (e.g., CIC-IoMT, CIC-UNSW-NB15) align better with sector-specific threats, whereas others, like CICIoV-24, underperform despite their recency. Our findings provide a standardized, interpretable approach for selecting datasets aligned with sector-specific operational requirements, ultimately enhancing the real-world effectiveness of AI-driven IDS/IPS deployments. The efficiency and practicality of the framework are validated through deployment in a real-world case study, underscoring its capacity to inform dataset selection and enhance the effectiveness of AI-driven IDS/IPS in operational environments.

翻译：基于机器学习（ML）与深度学习（DL）的入侵检测与防御系统（IDS/IPS）的性能，关键取决于训练与评估所用数据集的相关性与质量。然而，当前针对IDS/IPS开发的AI模型评估实践主要聚焦于准确性指标，往往忽略了数据集是否代表行业特定威胁。为弥补这一不足，我们提出了一种新颖的多维框架，该框架整合了用于威胁情报的MITRE ATT&CK知识库，并采用了五项互补性指标，共同提供对数据集适用性的全面评估。在方法论上，该框架结合威胁情报、自然语言处理与定量分析，以评估数据集对特定行业背景的适用性。将此框架应用于九个公开可用的IDS/IPS数据集，揭示了在威胁覆盖方面存在显著差距，尤其是在医疗保健、能源和金融领域。具体而言，较新的数据集（例如CIC-IoMT、CIC-UNSW-NB15）与行业特定威胁的契合度更高，而其他数据集，如CICIoV-24，尽管发布时间较近，但表现欠佳。我们的研究结果为选择符合行业特定操作需求的数据集提供了一种标准化、可解释的方法，最终提升了AI驱动的IDS/IPS部署在现实世界中的有效性。该框架的效率和实用性通过在一个真实案例研究中的部署得到了验证，突显了其在指导数据集选择以及提升AI驱动的IDS/IPS在操作环境中有效性方面的能力。