DETR-like models have significantly boosted the performance of detectors and even outperformed classical convolutional models. However, all tokens are treated equally without discrimination brings a redundant computational burden in the traditional encoder structure. The recent sparsification strategies exploit a subset of informative tokens to reduce attention complexity maintaining performance through the sparse encoder. But these methods tend to rely on unreliable model statistics. Moreover, simply reducing the token population hinders the detection performance to a large extent, limiting the application of these sparse models. We propose Focus-DETR, which focuses attention on more informative tokens for a better trade-off between computation efficiency and model accuracy. Specifically, we reconstruct the encoder with dual attention, which includes a token scoring mechanism that considers both localization and category semantic information of the objects from multi-scale feature maps. We efficiently abandon the background queries and enhance the semantic interaction of the fine-grained object queries based on the scores. Compared with the state-of-the-art sparse DETR-like detectors under the same setting, our Focus-DETR gets comparable complexity while achieving 50.4AP (+2.2) on COCO. The code is available at https://github.com/huawei-noah/noah-research/tree/master/Focus-DETR and https://gitee.com/mindspore/models/tree/master/research/cv/Focus-DETR.

翻译：DETR类模型显著提升了检测器的性能，甚至超越了经典卷积模型。然而，传统编码器结构中对所有令牌不加区分地同等处理，带来了冗余的计算负担。近期提出的稀疏化策略利用信息丰富的令牌子集来降低注意力复杂度，通过稀疏编码器维持性能。但这些方法往往依赖不可靠的模型统计数据。此外，简单减少令牌数量会严重阻碍检测性能，限制了这些稀疏模型的应用。我们提出Focus-DETR，通过聚焦注意力于信息更丰富的令牌，在计算效率与模型精度之间实现更优权衡。具体而言，我们利用双重注意力重构编码器，其中包含一个令牌评分机制，该机制综合考虑了多尺度特征图中目标的定位与类别语义信息。基于评分结果，我们有效舍弃背景查询，并增强细粒度目标查询的语义交互。在与最先进的稀疏DETR类检测器相同设置下的对比中，Focus-DETR在COCO数据集上以相当的复杂度实现了50.4AP（+2.2）。代码已开源：https://github.com/huawei-noah/noah-research/tree/master/Focus-DETR 及 https://gitee.com/mindspore/models/tree/master/research/cv/Focus-DETR。