In this paper, we examine computational approaches for measuring the "fairness" of image tagging systems, finding that they cluster into five distinct categories, each with its own analytic foundation. We also identify a range of normative concerns that are often collapsed under the terms "unfairness," "bias," or even "discrimination" when discussing problematic cases of image tagging. Specifically, we identify four types of representational harms that can be caused by image tagging systems, providing concrete examples of each. We then consider how different computational measurement approaches map to each of these types, demonstrating that there is not a one-to-one mapping. Our findings emphasize that no single measurement approach will be definitive and that it is not possible to infer from the use of a particular measurement approach which type of harm was intended to be measured. Lastly, equipped with this more granular understanding of the types of representational harms that can be caused by image tagging systems, we show that attempts to mitigate some of these types of harms may be in tension with one another.

翻译：本文研究了用于衡量图像标注系统“公平性”的计算方法，发现这些方法可分为五个不同类别，每类具有各自的分析基础。同时，在讨论图像标注的问题案例时，我们识别出常被“不公平性”“偏见”甚至“歧视”等术语笼统涵盖的一系列规范性关切。具体而言，我们界定了图像标注系统可能引发的四类表征危害，并为每类提供了具体实例。随后，我们探讨了不同的计算测量方法如何映射至各类危害，证明二者并非一一对应关系。研究结果表明，没有任何单一的测量方法具有决定性作用，且无法从使用特定测量方法推断出其所意图测量的危害类型。最后，基于对图像标注系统可能引发的表征危害类型的细化理解，我们发现缓解某些类型危害的努力可能彼此相互制约。