Recently, the RGB images and point clouds fusion methods have been proposed to jointly estimate 2D optical flow and 3D scene flow. However, as both conventional RGB cameras and LiDAR sensors adopt a frame-based data acquisition mechanism, their performance is limited by the fixed low sampling rates, especially in highly-dynamic scenes. By contrast, the event camera can asynchronously capture the intensity changes with a very high temporal resolution, providing complementary dynamic information of the observed scenes. In this paper, we incorporate RGB images, Point clouds and Events for joint optical flow and scene flow estimation with our proposed multi-stage multimodal fusion model, RPEFlow. First, we present an attention fusion module with a cross-attention mechanism to implicitly explore the internal cross-modal correlation for 2D and 3D branches, respectively. Second, we introduce a mutual information regularization term to explicitly model the complementary information of three modalities for effective multimodal feature learning. We also contribute a new synthetic dataset to advocate further research. Experiments on both synthetic and real datasets show that our model outperforms the existing state-of-the-art by a wide margin. Code and dataset is available at https://npucvr.github.io/RPEFlow.

翻译：近期，融合RGB图像与点云的方法已被提出用于联合估计二维光流与三维场景流。然而，由于传统RGB相机与激光雷达均采用基于帧的数据采集机制，其性能受限于固定的低采样率，尤其在高度动态场景中表现受限。相比之下，事件相机能够以极高时间分辨率异步捕捉强度变化，提供观测场景的补充动态信息。本文通过提出的多阶段多模态融合模型RPEFlow，融合RGB图像、点云与事件实现光流与场景流的联合估计。首先，我们提出一种采用交叉注意力机制的注意力融合模块，分别隐式探索二维与三维分支的跨模态内在关联。其次，引入互信息正则化项显式建模三种模态的互补信息，以实现有效的多模态特征学习。同时，我们贡献了一个新的合成数据集以推动后续研究。在合成与真实数据集上的实验表明，本模型以较大优势超越现有最优方法。代码与数据集发布于https://npucvr.github.io/RPEFlow。