Network delays, throughput bottlenecks and privacy issues push Artificial Intelligence of Things (AIoT) designers towards evaluating the feasibility of moving model training and execution (inference) as near as possible to the terminals. Meanwhile, results from the TinyML community demonstrate that, in some cases, it is possible to execute model inference directly on the terminals themselves, even if these are small microcontroller-based devices. However, to date, researchers and practitioners in the domain lack convenient all-in-one toolkits to help them evaluate the feasibility of moving execution of arbitrary models to arbitrary low-power IoT hardware. To this effect, we present in this paper U-TOE, a universal toolkit we designed to facilitate the task of AIoT designers and researchers, by combining functionalities from a low-power embedded OS, a generic model transpiler and compiler, an integrated performance measurement module, and an open-access remote IoT testbed. We provide an open source implementation of U-TOE and we demonstrate its use to experimentally evaluate the performance of a wide variety of models, on a wide variety of low-power boards, based on popular microcontroller architectures (ARM Cortex-M and RISC-V). U-TOE thus allows easily reproducible and customisable comparative evaluation experiments in this domain, on a wide variety of IoT hardware all-at-once. The availability of a toolkit such as U-TOE is desirable to accelerate the field of AIoT, towards fully exploiting the potential of edge computing.

翻译：网络延迟、吞吐量瓶颈及隐私问题，正推动人工智能物联网(AIoT)设计者评估将模型训练与执行（推理）尽可能迁移至终端侧的可行性。同时，TinyML社区的成果表明，在某些场景下，模型推理可直接在终端设备（即使是以微控制器为核心的小型装置）上执行。然而，该领域的研究者和实践者至今仍缺乏便捷的一体化工具包，用以评估将任意模型迁移至任意低功耗物联网硬件的可行性。为此，本文提出U-TOE——一种通用工具包，通过整合低功耗嵌入式操作系统、通用模型转译器与编译器、集成性能测量模块以及开放式远程物联网测试平台，专为辅助AIoT设计者与研究者而设计。我们提供了U-TOE的开源实现，并展示其如何基于主流微控制器架构（ARM Cortex-M与RISC-V），在各类低功耗开发板上实验评估多种模型的性能。U-TOE可支持该领域内易于复现且可自定义的对比评估实验，同时覆盖多种物联网硬件。此类工具包的普及对于加速AIoT领域发展、充分挖掘边缘计算潜力具有重要意义。