Vision Transformer (ViT) has recently gained significant interest in solving computer vision (CV) problems due to its capability of extracting informative features and modeling long-range dependencies through the self-attention mechanism. To fully realize the advantages of ViT in real-world applications, recent works have explored the trustworthiness of ViT, including its robustness and explainability. However, another desiderata, fairness has not yet been adequately addressed in the literature. We establish that the existing fairness-aware algorithms (primarily designed for CNNs) do not perform well on ViT. This necessitates the need for developing our novel framework via Debiased Self-Attention (DSA). DSA is a fairness-through-blindness approach that enforces ViT to eliminate spurious features correlated with the sensitive attributes for bias mitigation. Notably, adversarial examples are leveraged to locate and mask the spurious features in the input image patches. In addition, DSA utilizes an attention weights alignment regularizer in the training objective to encourage learning informative features for target prediction. Importantly, our DSA framework leads to improved fairness guarantees over prior works on multiple prediction tasks without compromising target prediction performance

翻译：视觉Transformer（ViT）近期因其通过自注意力机制提取信息特征和建模长程依赖的能力，在解决计算机视觉（CV）问题上引发了广泛关注。为在现实应用中充分发挥ViT的优势，已有研究探索了ViT的可信性，包括其鲁棒性和可解释性。然而，公平性这一重要期望在文献中尚未得到充分解决。我们证实，现有公平性感知算法（主要针对CNN设计）在ViT上表现不佳，这促使我们通过去偏自注意力（DSA）开发新型框架。DSA是一种"通过盲目实现公平"的方法，它强制ViT消除与敏感属性相关的虚假特征以实现偏差缓解。值得注意的是，该方法利用对抗样本定位并掩盖输入图像块中的虚假特征。此外，DSA在训练目标中引入注意力权重对齐正则化项，以鼓励学习用于目标预测的信息特征。重要的是，我们的DSA框架在多个预测任务上相比先前工作提供了更优的公平性保证，且不牺牲目标预测性能。