Event camera, as an emerging biologically-inspired vision sensor for capturing motion dynamics, presents new potential for 3D human pose tracking, or video-based 3D human pose estimation. However, existing works in pose tracking either require the presence of additional gray-scale images to establish a solid starting pose, or ignore the temporal dependencies all together by collapsing segments of event streams to form static event frames. Meanwhile, although the effectiveness of Artificial Neural Networks (ANNs, a.k.a. dense deep learning) has been showcased in many event-based tasks, the use of ANNs tends to neglect the fact that compared to the dense frame-based image sequences, the occurrence of events from an event camera is spatiotemporally much sparser. Motivated by the above mentioned issues, we present in this paper a dedicated end-to-end sparse deep learning approach for event-based pose tracking: 1) to our knowledge this is the first time that 3D human pose tracking is obtained from events only, thus eliminating the need of accessing to any frame-based images as part of input; 2) our approach is based entirely upon the framework of Spiking Neural Networks (SNNs), which consists of Spike-Element-Wise (SEW) ResNet and a novel Spiking Spatiotemporal Transformer; 3) a large-scale synthetic dataset is constructed that features a broad and diverse set of annotated 3D human motions, as well as longer hours of event stream data, named SynEventHPD. Empirical experiments demonstrate that, with superior performance over the state-of-the-art (SOTA) ANNs counterparts, our approach also achieves a significant computation reduction of 80% in FLOPS. Furthermore, our proposed method also outperforms SOTA SNNs in the regression task of human pose tracking. Our implementation is available at https://github.com/JimmyZou/HumanPoseTracking_SNN and dataset will be released upon paper acceptance.

翻译：事件相机作为一种新兴的仿生视觉传感器，能够捕捉运动动态信息，为三维人体姿态跟踪（即基于视频的三维人体姿态估计）带来了新的潜力。然而，现有姿态跟踪方法要么需要额外的灰度图像来建立可靠的初始姿态，要么通过压缩事件流片段形成静态事件帧而完全忽略时间依赖性。同时，尽管人工神经网络（ANNs，即密集深度学习）在诸多事件驱动任务中展现出有效性，但ANNs的使用往往忽视了这样一个事实：与密集的基于帧的图像序列相比，事件相机产生的事件在时空上要稀疏得多。受上述问题启发，本文提出了一种专用的端到端稀疏深度学习方法用于事件驱动姿态跟踪：1）据我们所知，这是首次仅凭事件数据实现三维人体姿态跟踪，从而消除了对任何基于帧的图像作为输入的需求；2）我们的方法完全基于脉冲神经网络（SNNs）框架，由逐元素脉冲残差网络（SEW ResNet）和一种新型脉冲时空Transformer构成；3）我们构建了一个大规模合成数据集SynEventHPD，其中包含广泛多样的带标注三维人体运动数据以及更长时段的脉冲流数据。实验证明，与最先进的ANNs方法相比，我们的方法不仅性能更优，而且计算量显著降低80%。此外，所提方法在人体姿态跟踪回归任务中也超越了最先进的SNNs。我们的实现代码可在https://github.com/JimmyZou/HumanPoseTracking_SNN获取，数据集将在论文接收后发布。