Window-based attention has become a popular choice in vision transformers due to its superior performance, lower computational complexity, and less memory footprint. However, the design of hand-crafted windows, which is data-agnostic, constrains the flexibility of transformers to adapt to objects of varying sizes, shapes, and orientations. To address this issue, we propose a novel quadrangle attention (QA) method that extends the window-based attention to a general quadrangle formulation. Our method employs an end-to-end learnable quadrangle regression module that predicts a transformation matrix to transform default windows into target quadrangles for token sampling and attention calculation, enabling the network to model various targets with different shapes and orientations and capture rich context information. We integrate QA into plain and hierarchical vision transformers to create a new architecture named QFormer, which offers minor code modifications and negligible extra computational cost. Extensive experiments on public benchmarks demonstrate that QFormer outperforms existing representative vision transformers on various vision tasks, including classification, object detection, semantic segmentation, and pose estimation. The code will be made publicly available at \href{https://github.com/ViTAE-Transformer/QFormer}{QFormer}.

翻译：基于窗口的注意力机制因其优越的性能、较低的计算复杂度和更少的内存占用，已成为视觉Transformer中的流行选择。然而，手工设计的窗口与数据无关，限制了Transformer适应不同尺寸、形状和方向物体的灵活性。为解决这一问题，我们提出了一种新颖的四边形注意力（QA）方法，将基于窗口的注意力扩展为通用的四边形形式。该方法采用端到端可学习的四边形回归模块，预测变换矩阵，将默认窗口转换为目标四边形以进行令牌采样和注意力计算，从而使网络能够建模具有不同形状和方向的各种目标，并捕获丰富的上下文信息。我们将QA集成到普通和分层视觉Transformer中，构建名为QFormer的新架构，该架构仅需少量代码修改且额外计算成本可忽略。在公共基准上的大量实验表明，QFormer在多种视觉任务（包括分类、目标检测、语义分割和姿态估计）上优于现有代表性视觉Transformer。代码将在https://github.com/ViTAE-Transformer/QFormer 公开。