As transformer architectures become increasingly prevalent in computer vision, it is critical to understand their fairness implications. We perform the first study of the fairness of transformers applied to computer vision and benchmark several bias mitigation approaches from prior work. We visualize the feature space of the transformer self-attention modules and discover that a significant portion of the bias is encoded in the query matrix. With this knowledge, we propose TADeT, a targeted alignment strategy for debiasing transformers that aims to discover and remove bias primarily from query matrix features. We measure performance using Balanced Accuracy and Standard Accuracy, and fairness using Equalized Odds and Balanced Accuracy Difference. TADeT consistently leads to improved fairness over prior work on multiple attribute prediction tasks on the CelebA dataset, without compromising performance.

翻译：随着Transformer架构在计算机视觉领域的日益普及，理解其公平性影响至关重要。我们首次对应用于计算机视觉的Transformer的公平性进行了研究，并基准测试了先前工作中的多种偏差缓解方法。通过可视化Transformer自注意力模块的特征空间，发现大部分偏差编码在查询矩阵中。基于此发现，我们提出TADeT——一种针对Transformer的去偏差目标对齐策略，旨在主要从查询矩阵特征中发现并移除偏差。我们使用平衡准确率和标准准确率衡量性能，使用均值机会差异和平衡准确率差值衡量公平性。在CelebA数据集的多个属性预测任务上，TADeT在保持性能不降低的情况下，持续提升公平性，优于先前方法。