Recent trackers adopt the Transformer to combine or replace the widely used ResNet as their new backbone network. Although their trackers work well in regular scenarios, however, they simply flatten the 2D features into a sequence to better match the Transformer. We believe these operations ignore the spatial prior of the target object which may lead to sub-optimal results only. In addition, many works demonstrate that self-attention is actually a low-pass filter, which is independent of input features or key/queries. That is to say, it may suppress the high-frequency component of the input features and preserve or even amplify the low-frequency information. To handle these issues, in this paper, we propose a unified Spatial-Frequency Transformer that models the Gaussian spatial Prior and High-frequency emphasis Attention (GPHA) simultaneously. To be specific, Gaussian spatial prior is generated using dual Multi-Layer Perceptrons (MLPs) and injected into the similarity matrix produced by multiplying Query and Key features in self-attention. The output will be fed into a Softmax layer and then decomposed into two components, i.e., the direct signal and high-frequency signal. The low- and high-pass branches are rescaled and combined to achieve all-pass, therefore, the high-frequency features will be protected well in stacked self-attention layers. We further integrate the Spatial-Frequency Transformer into the Siamese tracking framework and propose a novel tracking algorithm, termed SFTransT. The cross-scale fusion based SwinTransformer is adopted as the backbone, and also a multi-head cross-attention module is used to boost the interaction between search and template features. The output will be fed into the tracking head for target localization. Extensive experiments on both short-term and long-term tracking benchmarks all demonstrate the effectiveness of our proposed framework.

翻译：近期跟踪器采用Transformer结合或替代广泛使用的ResNet作为新的骨干网络。尽管这些跟踪器在常规场景下表现良好，但它们在将二维特征展平为序列以适配Transformer时，忽略了目标物体的空间先验信息，这可能导致仅获得次优结果。此外，许多研究表明自注意力机制本质上是低通滤波器，其特性与输入特征或键/查询无关，即它可能抑制输入特征的高频成分，同时保留甚至放大低频信息。为解决这些问题，本文提出统一的**空间-频率变换器**，同时建模高斯空间先验与高频强调注意力。具体而言，通过双多层感知机生成高斯空间先验，并将其注入自注意力中查询与键特征相乘得到的相似度矩阵。该输出经Softmax层后分解为两个分量：直通信道与高频信号通道。通过缩放并合并低通分支与高通分支实现全通特性，从而在堆叠的自注意力层中有效保护高频特征。我们将空间-频率变换器集成至孪生跟踪框架，提出新型跟踪算法SFTransT。采用基于跨尺度融合的SwinTransformer作为骨干网络，并利用多头交叉注意力模块增强搜索与模板特征间的交互，最终将输出输入跟踪头进行目标定位。在短期与长期跟踪基准上的大量实验均验证了所提框架的有效性。