Smart homes are powered by numerous programmable IoT platforms. Despite tremendous innovations, these platforms often suffer from safety and security issues. One class of defense solutions dynamically enforces safety and security policies, which essentially capture the expected behavior of the IoT system. While many proposed works were built on this runtime approach, they all are under-vetted. The primary reason lies in their evaluation approach. They are mostly self-evaluated in isolation using a virtual testbed combined with manually orchestrated test scenarios that rely on user interactions with the platform's UI. Such hand-crafted and non-uniform evaluation setups are limiting not only the reproducibility but also a comparative analysis of their efficacy results. Closing this gap in the traditional way requires a huge upfront manual effort, which causes the researchers turn away from any large-scale comparative empirical evaluation. Therefore, in this paper, we propose a highly-automated uniform evaluation platform, dubbed VetIoT, to vet the defense solutions that hinge on runtime policy enforcement. Given a defense solution, VetIoT easily instantiates a virtual testbed inside which the solution is empirically evaluated. VetIoT replaces manual UI-based interactions with an automated event simulator and manual inspection of test outcomes with an automated comparator. We developed a fully-functional prototype of VetIoT and applied it on three runtime policy enforcement solutions: Expat, Patriot, and IoTguard. VetIoT reproduced their individual prior results and assessed their efficacy results via stress testing and differential testing. We believe VetIoT can foster future research/evaluation.

翻译：智能家居由众多可编程物联网平台驱动。尽管创新层出不穷，这些平台仍常面临安全性与可靠性问题。一类防御方案通过动态执行安全与可靠性策略来应对风险，这些策略本质上刻画了物联网系统的预期行为。尽管诸多研究工作基于这种运行时方法构建，但均未得到充分验证。主要原因在于其评估方式：多数方案采用虚拟测试床结合手动编排的测试场景（依赖用户与平台图形界面交互）进行孤立自评。这种手工定制、非统一的评估设置不仅限制了结果的可复现性，更阻碍了防御效能的比较分析。传统方法若要弥合这一缺陷需投入大量前期人工工作，导致研究者放弃大规模比较性实证评估。为此，本文提出高度自动化的统一评测平台VetIoT，专门用于评估依赖运行时策略执行的防御方案。给定防御方案后，VetIoT可轻松实例化虚拟测试床，在该环境中进行实证评估。VetIoT用自动化事件模拟器替代手动图形界面交互，用自动比较器替代人工检查测试结果。我们开发了VetIoT的全功能原型，并将其应用于三种运行时策略执行方案：Expat、Patriot和IoTguard。VetIoT复现了各方案先前的独立实验结果，并通过压力测试与差异测试评估了其效能结果。我们相信VetIoT能够推动未来相关研究/评估工作的进展。