Vision Transformer (ViT) has recently gained significant interest in solving computer vision (CV) problems due to its capability of extracting informative features and modeling long-range dependencies through the self-attention mechanism. To fully realize the advantages of ViT in real-world applications, recent works have explored the trustworthiness of ViT, including its robustness and explainability. However, another desiderata, fairness has not yet been adequately addressed in the literature. We establish that the existing fairness-aware algorithms (primarily designed for CNNs) do not perform well on ViT. This necessitates the need for developing our novel framework via Debiased Self-Attention (DSA). DSA is a fairness-through-blindness approach that enforces ViT to eliminate spurious features correlated with the sensitive attributes for bias mitigation. Notably, adversarial examples are leveraged to locate and mask the spurious features in the input image patches. In addition, DSA utilizes an attention weights alignment regularizer in the training objective to encourage learning informative features for target prediction. Importantly, our DSA framework leads to improved fairness guarantees over prior works on multiple prediction tasks without compromising target prediction performance.

翻译：视觉Transformer（ViT）近年来因通过自注意力机制提取信息特征和建模长距离依赖的能力，在解决计算机视觉问题中引起了广泛关注。为充分实现ViT在实际应用中的优势，近期研究探讨了ViT的可信性，包括其鲁棒性和可解释性。然而，另一个关键需求——公平性，在现有文献中尚未得到充分解决。我们证明，现有的公平性感知算法（主要面向CNN设计）在ViT上表现不佳。这促使我们开发了一种基于去偏自注意力（DSA）的新框架。DSA是一种通过“盲视”实现公平性的方法，强制ViT消除与敏感属性相关的虚假特征以缓解偏差。值得注意的是，该方法利用对抗样本来定位并掩盖输入图像块中的虚假特征。此外，DSA在训练目标中引入注意力权重对齐正则化器，以鼓励学习用于目标预测的信息特征。重要的是，我们的DSA框架在多个预测任务中比先前工作提供了更好的公平性保证，且不影响目标预测性能。