Recently, vision Transformers (ViTs) have been actively applied to fine-grained visual recognition (FGVR). ViT can effectively model the interdependencies between patch-divided object regions through an inherent self-attention mechanism. In addition, patch selection is used with ViT to remove redundant patch information and highlight the most discriminative object patches. However, existing ViT-based FGVR models are limited to single-scale processing, and their fixed receptive fields hinder representational richness and exacerbate vulnerability to scale variability. Therefore, we propose multi-scale patch selection (MSPS) to improve the multi-scale capabilities of existing ViT-based models. Specifically, MSPS selects salient patches of different scales at different stages of a multi-scale vision Transformer (MS-ViT). In addition, we introduce class token transfer (CTT) and multi-scale cross-attention (MSCA) to model cross-scale interactions between selected multi-scale patches and fully reflect them in model decisions. Compared to previous single-scale patch selection (SSPS), our proposed MSPS encourages richer object representations based on feature hierarchy and consistently improves performance from small-sized to large-sized objects. As a result, we propose M2Former, which outperforms CNN-/ViT-based models on several widely used FGVR benchmarks.

翻译：近期，视觉Transformer（ViTs）已被积极应用于细粒度视觉识别（FGVR）。ViT通过固有的自注意力机制，能够有效建模经补丁划分的目标区域间的相互依赖关系。此外，结合ViT的补丁选择方法被用于去除冗余的补丁信息，并突出最具判别性的目标补丁。然而，现有基于ViT的FGVR模型局限于单尺度处理，其固定的感受野阻碍了表征丰富性，并加剧了对尺度变化的脆弱性。为此，我们提出多尺度补丁选择（MSPS）以提升现有基于ViT模型的多尺度能力。具体而言，MSPS在多尺度视觉Transformer（MS-ViT）的不同阶段选取不同尺度的显著补丁。此外，我们引入类别令牌传递（CTT）和多尺度交叉注意力（MSCA），对所选多尺度补丁之间的跨尺度交互进行建模，并使其在模型决策中得到充分体现。相较于先前的单尺度补丁选择（SSPS），我们提出的MSPS基于特征层次结构促进了更丰富的目标表征，并持续提升了从小到大型目标的识别性能。基于此，我们提出M2Former，其在多个广泛使用的FGVR基准测试中均优于基于CNN/ViT的模型。