Vision Transformers implement multi-head self-attention via stacking multiple attention blocks. The query, key, and value are often intertwined and generated within those blocks via a single, shared linear transformation. This paper explores the concept of disentangling the key from the query and value, and adopting a manifold representation for the key. Our experiments reveal that decoupling and endowing the key with a manifold structure can enhance the model's performance. Specifically, ViT-B exhibits a 0.87% increase in top-1 accuracy, while Swin-T sees a boost of 0.52% in top-1 accuracy on the ImageNet-1K dataset, with eight charts in the manifold key. Our approach also yields positive results in object detection and instance segmentation tasks on the COCO dataset. We establish that these performance gains are not merely due to the simplicity of adding more parameters and computations. Future research may investigate strategies for cutting the budget of such representations and aim for further performance improvements based on our findings.

翻译：视觉Transformer通过堆叠多个注意力块来实现多头自注意力机制。查询、键和值通常在这些块内通过单一的共享线性变换相互交织生成。本文探讨了将键与查询和值解耦，并采用流形表示键的概念。我们的实验表明，解耦并为键赋予流形结构能够提升模型性能。具体而言，在ImageNet-1K数据集上，当流形键包含八个图表时，ViT-B的top-1准确率提高了0.87%，Swin-T的top-1准确率提升了0.52%。我们的方法在COCO数据集的目标检测和实例分割任务中也取得了积极成果。我们证实这些性能提升并非仅仅源于增加参数和计算量的简单操作。未来研究可探索降低此类表示计算成本的策略，并基于我们的发现寻求进一步的性能改进。