While features of different scales are perceptually important to visual inputs, existing vision transformers do not yet take advantage of them explicitly. To this end, we first propose a cross-scale vision transformer, CrossFormer. It introduces a cross-scale embedding layer (CEL) and a long-short distance attention (LSDA). On the one hand, CEL blends each token with multiple patches of different scales, providing the self-attention module itself with cross-scale features. On the other hand, LSDA splits the self-attention module into a short-distance one and a long-distance counterpart, which not only reduces the computational burden but also keeps both small-scale and large-scale features in the tokens. Moreover, through experiments on CrossFormer, we observe another two issues that affect vision transformers' performance, i.e. the enlarging self-attention maps and amplitude explosion. Thus, we further propose a progressive group size (PGS) paradigm and an amplitude cooling layer (ACL) to alleviate the two issues, respectively. The CrossFormer incorporating with PGS and ACL is called CrossFormer++. Extensive experiments show that CrossFormer++ outperforms the other vision transformers on image classification, object detection, instance segmentation, and semantic segmentation tasks. The code will be available at: https://github.com/cheerss/CrossFormer.

翻译：不同尺度的特征对于视觉输入具有感知上的重要性，然而现有视觉Transformer尚未能明确利用这些特征。为此，我们首先提出一种跨尺度视觉Transformer——CrossFormer。它引入了跨尺度嵌入层（CEL）和长短距离注意力（LSDA）。一方面，CEL将每个词元与不同尺度的多个图像块融合，为自注意力模块本身提供跨尺度特征。另一方面，LSDA将自注意力模块分解为短距离注意力和长距离注意力两部分，这不仅降低了计算负担，同时保留了词元中的小尺度和大尺度特征。此外，通过对CrossFormer的实验，我们观察到影响视觉Transformer性能的另外两个问题，即自注意力图扩张和幅度爆炸。因此，我们进一步提出渐进式分组大小（PGS）范式和幅度冷却层（ACL）来分别缓解这两个问题。融入PGS和ACL的CrossFormer被称为CrossFormer++。大量实验表明，CrossFormer++在图像分类、目标检测、实例分割和语义分割任务中均优于其他视觉Transformer。代码将在https://github.com/cheerss/CrossFormer上开源。