Vision Transformers have shown promising progress in various object detection tasks, including monocular 2D/3D detection and surround-view 3D detection. However, when used in essential and classic stereo 3D object detection, directly adopting those surround-view Transformers leads to slow convergence and significant precision drops. We argue that one of the causes of this defect is that the surround-view Transformers do not consider the stereo-specific image correspondence information. In a surround-view system, the overlapping areas are small, and thus correspondence is not a primary issue. In this paper, we explore the model design of vision Transformers in stereo 3D object detection, focusing particularly on extracting and encoding the task-specific image correspondence information. To achieve this goal, we present TS3D, a Transformer-based Stereo-aware 3D object detector. In the TS3D, a Disparity-Aware Positional Encoding (DAPE) model is proposed to embed the image correspondence information into stereo features. The correspondence is encoded as normalized disparity and is used in conjunction with sinusoidal 2D positional encoding to provide the location information of the 3D scene. To extract enriched multi-scale stereo features, we propose a Stereo Reserving Feature Pyramid Network (SRFPN). The SRFPN is designed to reserve the correspondence information while fusing intra-scale and aggregating cross-scale stereo features. Our proposed TS3D achieves a 41.29% Moderate Car detection average precision on the KITTI test set and takes 88 ms to detect objects from each binocular image pair. It is competitive with advanced counterparts in terms of both precision and inference speed.

翻译：视觉Transformer在多种目标检测任务中取得了显著进展，包括单目2D/3D检测和环视3D检测。然而，当应用于基础且经典的立体三维目标检测时，直接采用这些环视Transformer会导致收敛缓慢和精度显著下降。我们认为，造成这一缺陷的原因之一是环视Transformer未考虑立体视觉特有的图像对应关系信息。在环视系统中，重叠区域较小，因此对应关系并非首要问题。本文探索了视觉Transformer在立体三维目标检测中的模型设计，特别关注如何提取和编码任务特定的图像对应关系信息。为此，我们提出TS3D——一种基于Transformer的立体感知三维目标检测器。在TS3D中，我们设计了视差感知位置编码（DAPE）模型，将图像对应关系信息嵌入立体特征中。对应关系被编码为归一化视差，并与正弦二维位置编码结合使用，以提供三维场景的位置信息。为了提取丰富的多尺度立体特征，我们提出了立体保留特征金字塔网络（SRFPN）。SRFPN旨在保留对应关系信息，同时融合尺度内特征并聚合跨尺度立体特征。我们的TS3D在KITTI测试集上实现了41.29%的中等汽车检测平均精度，每对双目图像检测目标仅需88毫秒。在精度和推理速度上均与先进方法具有竞争力。