Biological sensing and processing is asynchronous and sparse, leading to low-latency and energy-efficient perception and action. In robotics, neuromorphic hardware for event-based vision and spiking neural networks promises to exhibit similar characteristics. However, robotic implementations have been limited to basic tasks with low-dimensional sensory inputs and motor actions due to the restricted network size in current embedded neuromorphic processors and the difficulties of training spiking neural networks. Here, we present the first fully neuromorphic vision-to-control pipeline for controlling a freely flying drone. Specifically, we train a spiking neural network that accepts high-dimensional raw event-based camera data and outputs low-level control actions for performing autonomous vision-based flight. The vision part of the network, consisting of five layers and 28.8k neurons, maps incoming raw events to ego-motion estimates and is trained with self-supervised learning on real event data. The control part consists of a single decoding layer and is learned with an evolutionary algorithm in a drone simulator. Robotic experiments show a successful sim-to-real transfer of the fully learned neuromorphic pipeline. The drone can accurately follow different ego-motion setpoints, allowing for hovering, landing, and maneuvering sideways$\unicode{x2014}$even while yawing at the same time. The neuromorphic pipeline runs on board on Intel's Loihi neuromorphic processor with an execution frequency of 200 Hz, spending only 27 $\unicode{x00b5}$J per inference. These results illustrate the potential of neuromorphic sensing and processing for enabling smaller, more intelligent robots.

翻译：生物感知与处理具有异步性和稀疏性，可实现低延迟、高能效的感知与行动。在机器人领域，基于事件视觉和脉冲神经网络的神经形态硬件有望展现类似特性。然而，由于当前嵌入式神经形态处理器的网络规模受限，以及脉冲神经网络训练的困难，机器人实现仅限于低维感官输入和电机动作的基本任务。本文首次提出用于控制自由飞行无人机的全神经形态视觉-控制流水线。具体而言，我们训练了一个脉冲神经网络，该网络接受高维原始事件相机数据，并输出底层控制指令以执行自主视觉飞行。网络的视觉部分由五层共28,800个神经元组成，将输入的原始事件映射为自运动估计，并采用真实事件数据的自监督学习进行训练。控制部分由单一解码层构成，通过无人机模拟器中的进化算法学习。机器人实验表明，该完全习得的神经形态流水线成功实现了从仿真到现实的迁移。无人机能精确跟踪不同的自运动设定点，实现悬停、降落和侧向机动——甚至同步偏航。该神经形态流水线在英特尔Loihi神经形态处理器上以200 Hz的执行频率运行，每次推理仅消耗27 µJ能量。这些结果证明了神经形态感知与处理在实现更小、更智能机器人方面的潜力。