Efficient on-device neural network (NN) inference has various advantages over cloud-based processing, including predictable latency, enhanced privacy, greater reliability, and reduced operating costs for vendors. This has sparked the recent rapid development of microcontroller-scale NN accelerators, often referred to as neural processing units ($\mu$NPUs), designed specifically for ultra-low-power applications. In this paper we present the first comparative evaluation of a number of commercially-available $\mu$NPUs, as well as the first independent benchmarks for several of these platforms. We develop and open-source a model compilation framework to enable consistent benchmarking of quantized models across diverse $\mu$NPU hardware. Our benchmark targets end-to-end performance and includes model inference latency, power consumption, and memory overhead, alongside other factors. The resulting analysis uncovers both expected performance trends as well as surprising disparities between hardware specifications and actual performance, including $\mu$NPUs exhibiting unexpected scaling behaviors with increasing model complexity. Our framework provides a foundation for further evaluation of $\mu$NPU platforms alongside valuable insights for both hardware designers and software developers in this rapidly evolving space.

翻译：相较于基于云端的处理，高效的设备端神经网络推理具有多种优势，包括可预测的延迟、增强的隐私性、更高的可靠性以及降低供应商的运营成本。这推动了近期微控制器级神经网络加速器（通常称为神经处理单元（$μ$NPUs））的快速发展，这些加速器专为超低功耗应用而设计。本文首次对多款商用$μ$NPU进行了比较性评估，并对其中数个平台提供了首批独立基准测试结果。我们开发并开源了一个模型编译框架，以实现跨不同$μ$NPU硬件对量化模型进行一致的基准测试。我们的基准测试关注端到端性能，包括模型推理延迟、功耗和内存开销以及其他因素。最终的分析揭示了预期的性能趋势，以及硬件规格与实际性能之间令人惊讶的差异，包括某些$μ$NPU在模型复杂度增加时表现出意料之外的扩展行为。我们的框架为进一步评估$μ$NPU平台奠定了基础，同时为这个快速发展的领域中的硬件设计者和软件开发者提供了宝贵的见解。