Object detection on visible (RGB) and infrared (IR) images, as an emerging solution to facilitate robust detection for around-the-clock applications, has received extensive attention in recent years. With the help of IR images, object detectors have been more reliable and robust in practical applications by using RGB-IR combined information. However, existing methods still suffer from modality miscalibration and fusion imprecision problems. Since transformer has the powerful capability to model the pairwise correlations between different features, in this paper, we propose a novel Calibrated and Complementary Transformer called $\mathrm{C}^2$Former to address these two problems simultaneously. In $\mathrm{C}^2$Former, we design an Inter-modality Cross-Attention (ICA) module to obtain the calibrated and complementary features by learning the cross-attention relationship between the RGB and IR modality. To reduce the computational cost caused by computing the global attention in ICA, an Adaptive Feature Sampling (AFS) module is introduced to decrease the dimension of feature maps. Because $\mathrm{C}^2$Former performs in the feature domain, it can be embedded into existed RGB-IR object detectors via the backbone network. Thus, one single-stage and one two-stage object detector both incorporating our $\mathrm{C}^2$Former are constructed to evaluate its effectiveness and versatility. With extensive experiments on the DroneVehicle and KAIST RGB-IR datasets, we verify that our method can fully utilize the RGB-IR complementary information and achieve robust detection results. The code is available at https://github.com/yuanmaoxun/Calibrated-and-Complementary-Transformer-for-RGB-Infrared-Object-Detection.git.

翻译：可见光（RGB）与红外（IR）图像的目标检测，作为实现全天候场景下鲁棒检测的新兴解决方案，近年来受到广泛关注。借助红外图像的辅助，目标检测器通过融合RGB-IR联合信息，在实际应用中表现出更高的可靠性和鲁棒性。然而，现有方法仍存在模态校准偏差与融合精度不足的问题。由于Transformer具有强大的特征间成对相关性建模能力，本文提出一种名为$\mathrm{C}^2$Former的新型校准与互补Transformer，以同时解决上述两个问题。在$\mathrm{C}^2$Former中，我们设计了模态间交叉注意力模块（ICA），通过学习RGB与IR模态间的交叉注意力关系来获取校准且互补的特征。为降低ICA中全局注意力计算带来的计算开销，引入自适应特征采样模块（AFS）以减小特征图维度。由于$\mathrm{C}^2$Former在特征域中执行，可通过主干网络嵌入现有RGB-IR目标检测器。据此，我们分别构建了集成$\mathrm{C}^2$Former的单阶段与双阶段目标检测器，以评估其有效性与通用性。在DroneVehicle和KAIST RGB-IR数据集上的大量实验表明，该方法能充分利用RGB-IR互补信息，实现鲁棒的检测结果。代码见https://github.com/yuanmaoxun/Calibrated-and-Complementary-Transformer-for-RGB-Infrared-Object-Detection.git。