Visual sensors, including 3D LiDAR, neuromorphic DVS sensors, and conventional frame cameras, are increasingly integrated into edge-side intelligent machines. Realizing intensive multi-sensory data analysis directly on edge intelligent machines is crucial for numerous emerging edge applications, such as augmented and virtual reality and unmanned aerial vehicles, which necessitates unified data representation, unprecedented hardware energy efficiency and rapid model training. However, multi-sensory data are intrinsically heterogeneous, causing significant complexity in the system development for edge-side intelligent machines. In addition, the performance of conventional digital hardware is limited by the physically separated processing and memory units, known as the von Neumann bottleneck, and the physical limit of transistor scaling, which contributes to the slowdown of Moore's law. These limitations are further intensified by the tedious training of models with ever-increasing sizes. We propose a novel hardware-software co-design, random resistive memory-based deep extreme point learning machine (DEPLM), that offers efficient unified point set analysis. We show the system's versatility across various data modalities and two different learning tasks. Compared to a conventional digital hardware-based system, our co-design system achieves huge energy efficiency improvements and training cost reduction when compared to conventional systems. Our random resistive memory-based deep extreme point learning machine may pave the way for energy-efficient and training-friendly edge AI across various data modalities and tasks.

翻译：视觉传感器，包括3D激光雷达、神经形态DVS传感器和传统帧相机，正越来越多地集成到边缘智能机器中。直接在边缘智能机器上实现密集的多感官数据分析，对于增强现实、虚拟现实以及无人机等众多新兴边缘应用至关重要，这需要统一的数据表示、前所未有的硬件能效和快速的模型训练。然而，多感官数据本质上是异质的，导致边缘智能机器系统开发复杂度显著增加。此外，传统数字硬件的性能受到物理分离的处理单元和存储单元（即冯·诺依曼瓶颈）以及晶体管缩放物理极限的限制，这导致了摩尔定律的放缓。这些局限性因模型规模不断增长且训练过程繁琐而进一步加剧。我们提出了一种新颖的软硬件协同设计——基于随机电阻式存储器的深度极限点学习机（DEPLM），实现了高效统一的点集分析。我们展示了该系统在多种数据模态和两种不同学习任务中的通用性。与基于传统数字硬件的系统相比，我们的协同设计系统在能效提升和训练成本降低方面取得了巨大优势。我们的基于随机电阻式存储器的深度极限点学习机可能为跨多种数据模态和任务的节能且易于训练的边缘人工智能铺平道路。