This paper presents an investigation into the estimation of optical and scene flow using RGBD information in scenarios where the RGB modality is affected by noise or captured in dark environments. Existing methods typically rely solely on RGB images or fuse the modalities at later stages, which can result in lower accuracy when the RGB information is unreliable. To address this issue, we propose a novel deep neural network approach named FusionRAFT, which enables early-stage information fusion between sensor modalities (RGB and depth). Our approach incorporates self- and cross-attention layers at different network levels to construct informative features that leverage the strengths of both modalities. Through comparative experiments, we demonstrate that our approach outperforms recent methods in terms of performance on the synthetic dataset Flyingthings3D, as well as the generalization on the real-world dataset KITTI. We illustrate that our approach exhibits improved robustness in the presence of noise and low-lighting conditions that affect the RGB images. We release the code, models and dataset at https://github.com/jiesico/FusionRAFT.

翻译：本文针对RGBD信息中RGB模态受噪声影响或在暗环境捕获场景下的光学流与场景流估计问题展开研究。现有方法通常仅依赖RGB图像或在后期阶段进行模态融合，当RGB信息不可靠时会导致精度下降。为解决该问题，我们提出了一种名为FusionRAFT的新型深度神经网络方法，该方法实现了传感器模态（RGB与深度）之间的早期信息融合。本方法在不同网络层级引入自注意力与交叉注意力层，从而构建能够利用两种模态优势的信息特征。通过对比实验，我们证明该方法在合成数据集Flyingthings3D上的性能表现以及真实数据集KITTI上的泛化能力均优于近期方法。实验表明，本方法在影响RGB图像的噪声和低光照条件下展现出更强的鲁棒性。我们将代码、模型及数据集开源至https://github.com/jiesico/FusionRAFT。