Automated machine learning (AutoML) has emerged as a promising paradigm for automating machine learning (ML) pipeline design, broadening AI adoption. Yet its reliability in complex domains such as cybersecurity remains underexplored. This paper systematically evaluates eight open-source AutoML frameworks across 11 publicly available cybersecurity datasets, spanning intrusion detection, malware classification, phishing, fraud detection, and spam filtering. Results show substantial performance variability across tools and datasets, with no single solution consistently superior. A paradigm shift is observed: the challenge has moved from selecting individual ML models to identifying the most suitable AutoML framework, complicated by differences in runtime efficiency, automation capabilities, and supported features. AutoML tools frequently favor tree-based models, which perform well but risk overfitting and limit interpretability. Key challenges identified include adversarial vulnerability, model drift, and inadequate feature engineering. We conclude with best practices and research directions to strengthen robustness, interpretability, and trust in AutoML for high-stakes cybersecurity applications.

翻译：自动化机器学习（AutoML）作为一种自动化机器学习（ML）流程设计的范式，已展现出广阔前景，并推动了人工智能的更广泛采用。然而，其在网络安全等复杂领域的可靠性仍未得到充分探索。本文系统评估了八个开源AutoML框架在11个公开可用的网络安全数据集上的表现，涵盖入侵检测、恶意软件分类、网络钓鱼、欺诈检测和垃圾邮件过滤等任务。结果显示，不同工具和数据集之间存在显著的性能差异，没有单一的解决方案能始终保持最优。我们观察到一个范式转变：挑战已从选择单个ML模型转变为识别最合适的AutoML框架，而这一过程因运行效率、自动化能力和支持功能等方面的差异而变得复杂。AutoML工具通常倾向于选择基于树的模型，这些模型性能良好但存在过拟合风险，并限制了模型的可解释性。研究识别出的关键挑战包括对抗性漏洞、模型漂移以及特征工程不足。最后，我们提出了最佳实践和研究方向，以增强AutoML在高风险网络安全应用中的鲁棒性、可解释性和可信度。