Machine learning-based supervised classifiers are widely used for security tasks, and their improvement has been largely focused on algorithmic advancements. We argue that data challenges that negatively impact the performance of these classifiers have received limited attention. We address the following research question: Can developments in Generative AI (GenAI) address these data challenges and improve classifier performance? We propose augmenting training datasets with synthetic data generated using GenAI techniques to improve classifier generalization. We evaluate this approach across 7 diverse security tasks using 6 state-of-the-art GenAI methods and introduce a novel GenAI scheme called Nimai that enables highly controlled data synthesis. We find that GenAI techniques can significantly improve the performance of security classifiers, achieving improvements of up to 32.6% even in severely data-constrained settings (only ~180 training samples). Furthermore, we demonstrate that GenAI can facilitate rapid adaptation to concept drift post-deployment, requiring minimal labeling in the adjustment process. Despite successes, our study finds that some GenAI schemes struggle to initialize (train and produce data) on certain security tasks. We also identify characteristics of specific tasks, such as noisy labels, overlapping class distributions, and sparse feature vectors, which hinder performance boost using GenAI. We believe that our study will drive the development of future GenAI tools designed for security tasks.

翻译：基于机器学习的监督分类器广泛用于安防任务，其性能提升主要集中于算法改进。我们认为，对分类器性能产生负面影响的"数据挑战"问题尚未得到足够重视。本文探讨的核心研究问题是：生成式AI的发展能否解决这些数据挑战并提升分类器性能？我们提出通过生成式AI技术合成数据来扩充训练数据集，以改善分类器的泛化能力。我们采用6种前沿生成式AI方法，在7个不同安防任务中评估该方案，并引入一种名为"Nimai"的新型生成式AI框架，该框架可实现高度可控的数据合成。研究发现，生成式AI技术能显著提升安防分类器性能——即使在数据严重受限（仅约180个训练样本）的场景下，性能提升最高可达32.6%。此外，我们证明生成式AI能促进部署后对概念漂移的快速适应，调整过程中仅需极少量标注。尽管取得这些进展，本研究仍发现部分生成式AI方案在特定安防任务上难以完成初始化（训练与数据生成）。我们还识别出阻碍生成式AI性能提升的任务特征，例如标签噪声、类别分布重叠及特征向量稀疏性。我们认为，本研究成果将推动面向安防任务的下一代生成式AI工具研发。