Smart home IoT platforms such as openHAB rely on Trigger Action Condition (TAC) rules to automate device behavior, but the interplay among these rules can give rise to interaction threats, unintended or unsafe behaviors emerging from implicit dependencies, conflicting triggers, or overlapping conditions. Identifying these threats requires semantic understanding and structural reasoning that traditionally depend on symbolic, constraint-driven static analysis. This work presents the first comprehensive evaluation of Large Language Models (LLMs) across a multi-category interaction threat taxonomy, assessing their performance on both the original openHAB (oHC/IoTB) dataset and a structurally challenging Mutation dataset designed to test robustness under rule transformations. We benchmark Llama 3.1 8B, Llama 70B, GPT-4o, Gemini-2.5-Pro, and DeepSeek-R1 across zero-, one-, and two-shot settings, comparing their results against oHIT's manually validated ground truth. Our findings show that while LLMs exhibit promising semantic understanding, particularly on action- and condition-related threats, their accuracy degrades significantly for threats requiring cross-rule structural reasoning, especially under mutated rule forms. Model performance varies widely across threat categories and prompt settings, with no model providing consistent reliability. In contrast, the symbolic reasoning baseline maintains stable detection across both datasets, unaffected by rule rewrites or structural perturbations. These results underscore that LLMs alone are not yet dependable for safety critical interaction-threat detection in IoT environments. We discuss the implications for tool design and highlight the potential of hybrid architectures that combine symbolic analysis with LLM-based semantic interpretation to reduce false positives while maintaining structural rigor.

翻译：开放家庭自动化总线（openHAB）等智能家居物联网平台依赖触发-动作-条件（TAC）规则来实现设备行为的自动化，但这些规则之间的相互作用可能引发交互威胁，即由隐式依赖、冲突触发器或重叠条件产生的非预期或不安全行为。识别这些威胁需要语义理解和结构推理，传统上依赖于符号化、约束驱动的静态分析。本研究首次对大语言模型（LLMs）在多类别交互威胁分类体系上的表现进行了全面评估，测试了其在原始openHAB（oHC/IoTB）数据集以及一个旨在检验规则变换下鲁棒性的、结构上更具挑战性的变异数据集上的性能。我们以零样本、单样本和双样本设置对Llama 3.1 8B、Llama 70B、GPT-4o、Gemini-2.5-Pro和DeepSeek-R1进行了基准测试，并将其结果与oHIT人工验证的基准真值进行了比较。我们的研究结果表明，尽管LLMs展现出有前景的语义理解能力，尤其是在涉及动作和条件的威胁类别上，但对于需要跨规则结构推理的威胁，其准确性显著下降，在变异规则形式下尤为明显。不同模型在威胁类别和提示设置下的表现差异很大，没有模型能提供一致的可靠性。相比之下，基于符号推理的基线方法在两个数据集上均保持了稳定的检测能力，且不受规则重写或结构扰动的影响。这些结果强调，在物联网环境中，仅依赖LLMs尚不足以成为安全关键型交互威胁检测的可信赖方案。我们讨论了其对工具设计的启示，并强调了将符号分析与基于LLM的语义解释相结合的混合架构的潜力，这种架构可在保持结构严谨性的同时减少误报。