Detecting the objects in dense and rotated scenes is a challenging task. Recent works on this topic are mostly based on Faster RCNN or Retinanet. As they are highly dependent on the pre-set dense anchors and the NMS operation, the approach is indirect and suboptimal.The end-to-end DETR-based detectors have achieved great success in horizontal object detection and many other areas like segmentation, tracking, action recognition and etc.However, the DETR-based detectors perform poorly on dense rotated target tasks and perform worse than most modern CNN-based detectors. In this paper, we find the most significant reason for the poor performance is that the original attention can not accurately focus on the oriented targets. Accordingly, we propose Rotated object detection TRansformer (RotaTR) as an extension of DETR to oriented detection. Specifically, we design Rotation Sensitive deformable (RSDeform) attention to enhance the DETR's ability to detect oriented targets. It is used to build the feature alignment module and rotation-sensitive decoder for our model. We test RotaTR on four challenging-oriented benchmarks. It shows a great advantage in detecting dense and oriented objects compared to the original DETR. It also achieves competitive results when compared to the state-of-the-art.

翻译：在密集和旋转场景中检测目标是一项具有挑战性的任务。近期相关工作大多基于Faster RCNN或Retinanet。由于这些方法高度依赖预设的密集锚框和NMS操作，其实现间接且非最优。基于端到端DETR的检测器在水平目标检测以及分割、跟踪、动作识别等多个领域已取得显著成功。然而，DETR类检测器在密集旋转目标任务上表现不佳，性能逊于大多数现代CNN类检测器。本文发现，导致性能不足的最关键原因是原始注意力机制无法精确聚焦于有向目标。为此，我们提出旋转目标检测Transformer（RotaTR）作为DETR在有向检测领域的扩展。具体而言，我们设计了旋转敏感可变形（RSDeform）注意力机制以增强DETR检测有向目标的能力，并将其用于构建特征对齐模块和旋转敏感解码器。我们在四个具有挑战性的有向检测基准上测试RotaTR，结果表明，相比原始DETR，该方法在检测密集有向目标方面具有显著优势，同时在与现有最优方法对比时也取得了具有竞争力的结果。