Event camera-based pattern recognition is a newly arising research topic in recent years. Current researchers usually transform the event streams into images, graphs, or voxels, and adopt deep neural networks for event-based classification. Although good performance can be achieved on simple event recognition datasets, however, their results may be still limited due to the following two issues. Firstly, they adopt spatial sparse event streams for recognition only, which may fail to capture the color and detailed texture information well. Secondly, they adopt either Spiking Neural Networks (SNN) for energy-efficient recognition with suboptimal results, or Artificial Neural Networks (ANN) for energy-intensive, high-performance recognition. However, seldom of them consider achieving a balance between these two aspects. In this paper, we formally propose to recognize patterns by fusing RGB frames and event streams simultaneously and propose a new RGB frame-event recognition framework to address the aforementioned issues. The proposed method contains four main modules, i.e., memory support Transformer network for RGB frame encoding, spiking neural network for raw event stream encoding, multi-modal bottleneck fusion module for RGB-Event feature aggregation, and prediction head. Due to the scarce of RGB-Event based classification dataset, we also propose a large-scale PokerEvent dataset which contains 114 classes, and 27102 frame-event pairs recorded using a DVS346 event camera. Extensive experiments on two RGB-Event based classification datasets fully validated the effectiveness of our proposed framework. We hope this work will boost the development of pattern recognition by fusing RGB frames and event streams. Both our dataset and source code of this work will be released at https://github.com/Event-AHU/SSTFormer.

翻译：基于事件相机的模式识别是近年兴起的研究方向。当前研究者通常将事件流转换为图像、图结构或体素表示，并采用深度神经网络进行事件分类。尽管在简单事件识别数据集上可取得良好性能，但其结果仍受限于以下两个问题：其一，仅采用空间稀疏事件流进行识别，难以充分捕捉色彩与精细纹理信息；其二，现有方法或采用脉冲神经网络（SNN）实现低能耗但次优的识别，或采用人工神经网络（ANN）实现高能耗的高性能识别，鲜有研究能在两者间取得平衡。本文正式提出融合RGB帧与事件流进行模式识别的新方法，并构建了RGB-事件协同识别框架以解决上述问题。所提方法包含四个核心模块：RGB帧编码的记忆支持Transformer网络、原始事件流编码的脉冲神经网络、RGB-事件特征聚合的多模态瓶颈融合模块，以及预测头。针对RGB-事件分类数据集的稀缺现状，我们还构建了大型PokerEvent数据集，包含使用DVS346事件相机记录的114个类别、27102组帧-事件对。在两个RGB-事件分类数据集上的大量实验充分验证了所提框架的有效性。我们期望这项工作能推动融合RGB帧与事件流的模式识别技术发展。相关数据集与源代码将在https://github.com/Event-AHU/SSTFormer公开。