In the current demand for automation in the agro-food industry, accurately detecting and localizing relevant objects in 3D is essential for successful robotic operations. However, this is a challenge due the presence of occlusions. Multi-view perception approaches allow robots to overcome occlusions, but a tracking component is needed to associate the objects detected by the robot over multiple viewpoints. Most multi-object tracking (MOT) algorithms are designed for high frame rate sequences and struggle with the occlusions generated by robots' motions and 3D environments. In this paper, we introduce MOT-DETR, a novel approach to detect and track objects in 3D over time using a combination of convolutional networks and transformers. Our method processes 2D and 3D data, and employs a transformer architecture to perform data fusion. We show that MOT-DETR outperforms state-of-the-art multi-object tracking methods. Furthermore, we prove that MOT-DETR can leverage 3D data to deal with long-term occlusions and large frame-to-frame distances better than state-of-the-art methods. Finally, we show how our method is resilient to camera pose noise that can affect the accuracy of point clouds. The implementation of MOT-DETR can be found here: https://github.com/drapado/mot-detr

翻译：在当前农业食品行业自动化的需求下，准确检测并定位三维空间中的相关目标对于机器人成功执行操作至关重要。然而，由于遮挡的存在，这一任务面临挑战。多视角感知方法可使机器人克服遮挡问题，但需要跟踪组件来关联机器人在多视角下检测到的目标。绝大多数多目标跟踪（MOT）算法专为高帧率序列设计，难以应对机器人运动及三维环境产生的遮挡。本文提出MOT-DETR——一种结合卷积网络与Transformer的新型方法，用于随时间推移检测并跟踪三维空间中的目标。该方法处理二维与三维数据，并采用Transformer架构实现数据融合。实验表明，MOT-DETR的性能优于当前最先进的多目标跟踪方法。此外，我们证明MOT-DETR能比现有方法更有效地利用三维数据处理长期遮挡及大帧间位移问题。最后，我们展示了该方法对影响点云精度的相机位姿噪声具有鲁棒性。MOT-DETR的实现代码见：https://github.com/drapado/mot-detr