The Internet of Things has expanded rapidly, transforming communication and operations across industries but also increasing the attack surface and security breaches. Artificial Intelligence plays a key role in securing IoT, enabling attack detection, attack behavior analysis, and mitigation suggestion. Despite advancements, evaluations remain purely qualitative, and the lack of a standardized, objective benchmark for quantitatively measuring AI-based attack analysis and mitigation hinders consistent assessment of model effectiveness. In this work, we propose a hybrid framework combining Machine Learning (ML) for multi-class attack detection with Large Language Models (LLMs) for attack behavior analysis and mitigation suggestion. After benchmarking several ML and Deep Learning (DL) classifiers on the Edge-IIoTset and CICIoT2023 datasets, we applied structured role-play prompt engineering with Retrieval-Augmented Generation (RAG) to guide ChatGPT-o3 and DeepSeek-R1 in producing detailed, context-aware responses. We introduce novel evaluation metrics for quantitative assessment to guide us and an ensemble of judge LLMs, namely ChatGPT-4o, DeepSeek-V3, Mixtral 8x7B Instruct, Gemini 2.5 Flash, Meta Llama 4, TII Falcon H1 34B Instruct, xAI Grok 3, and Claude 4 Sonnet, to independently evaluate the responses. Results show that Random Forest has the best detection model, and ChatGPT-o3 outperformed DeepSeek-R1 in attack analysis and mitigation.

翻译：物联网已迅速扩展，改变了各行业的通信与运营方式，但也扩大了攻击面并增加了安全漏洞。人工智能在保障物联网安全方面发挥着关键作用，能够实现攻击检测、攻击行为分析和缓解建议。尽管取得了进展，现有评估仍停留在纯定性层面，缺乏用于定量衡量基于人工智能的攻击分析与缓解的标准化客观基准，这阻碍了对模型有效性的持续评估。在本研究中，我们提出了一种混合框架，将用于多类攻击检测的机器学习与用于攻击行为分析和缓解建议的大语言模型相结合。在Edge-IIoTset和CICIoT2023数据集上对多种机器学习和深度学习分类器进行基准测试后，我们采用结合检索增强生成的结构化角色扮演提示工程，引导ChatGPT-o3和DeepSeek-R1生成详细且具有上下文感知的响应。我们引入了新颖的定量评估指标，以指导我们及一组评审大语言模型——包括ChatGPT-4o、DeepSeek-V3、Mixtral 8x7B Instruct、Gemini 2.5 Flash、Meta Llama 4、TII Falcon H1 34B Instruct、xAI Grok 3和Claude 4 Sonnet——对响应进行独立评估。结果表明，随机森林具有最佳检测性能，而ChatGPT-o3在攻击分析与缓解方面优于DeepSeek-R1。