Scaled dot-product attention applies a softmax function on the scaled dot-product of queries and keys to calculate weights and then multiplies the weights and values. In this work, we study how to improve the learning of scaled dot-product attention to improve the accuracy of DETR. Our method is based on the following observations: using ground truth foreground-background mask (GT Fg-Bg Mask) as additional cues in the weights/values learning enables learning much better weights/values; with better weights/values, better values/weights can be learned. We propose a triple-attention module in which the first attention is a plain scaled dot-product attention, the second/third attention generates high-quality weights/values (with the assistance of GT Fg-Bg Mask) and shares the values/weights with the first attention to improve the quality of values/weights. The second and third attentions are removed during inference. We call our method knowledge-sharing DETR (KS-DETR), which is an extension of knowledge distillation (KD) in the way that the improved weights and values of the teachers (the second and third attentions) are directly shared, instead of mimicked, by the student (the first attention) to enable more efficient knowledge transfer from the teachers to the student. Experiments on various DETR-like methods show consistent improvements over the baseline methods on the MS COCO benchmark. Code is available at https://github.com/edocanonymous/KS-DETR.

翻译：缩放点积注意力通过对查询与键的缩放点积应用softmax函数计算权重，再将权重与值相乘。本研究旨在改进缩放点积注意力的学习方式，以提升DETR的检测精度。我们的方法基于以下观察：在权重/值学习过程中，使用真实前景-背景掩码（GT Fg-Bg Mask）作为额外线索，能够学习到更优的权重/值；而更优的权重/值又能促进更优的值/权重学习。我们提出三重注意力模块：第一重注意力为常规缩放点积注意力，第二/三重注意力在GT Fg-Bg Mask辅助下生成高质量权重/值，并与第一重注意力共享值/权重以提升后者的质量。推理阶段移除第二/三重注意力。我们将该方法命名为知识共享DETR（KS-DETR），该方法通过教师网络（第二/三重注意力）直接向学生网络（第一重注意力）共享改进后的权重/值，而非模仿教师输出，从而实现更高效的知识迁移。在多种类DETR方法上的实验表明，该方法在MS COCO基准测试中持续优于基线方法。代码发布于：https://github.com/edocanonymous/KS-DETR。