Most pre-trained learning systems are known to suffer from bias, which typically emerges from the data, the model, or both. Measuring and quantifying bias and its sources is a challenging task and has been extensively studied in image captioning. Despite the significant effort in this direction, we observed that existing metrics lack consistency in the inclusion of the visual signal. In this paper, we introduce a new bias assessment metric, dubbed $ImageCaptioner^2$, for image captioning. Instead of measuring the absolute bias in the model or the data, $ImageCaptioner^2$ pay more attention to the bias introduced by the model w.r.t the data bias, termed bias amplification. Unlike the existing methods, which only evaluate the image captioning algorithms based on the generated captions only, $ImageCaptioner^2$ incorporates the image while measuring the bias. In addition, we design a formulation for measuring the bias of generated captions as prompt-based image captioning instead of using language classifiers. Finally, we apply our $ImageCaptioner^2$ metric across 11 different image captioning architectures on three different datasets, i.e., MS-COCO caption dataset, Artemis V1, and Artemis V2, and on three different protected attributes, i.e., gender, race, and emotions. Consequently, we verify the effectiveness of our $ImageCaptioner^2$ metric by proposing AnonymousBench, which is a novel human evaluation paradigm for bias metrics. Our metric shows significant superiority over the recent bias metric; LIC, in terms of human alignment, where the correlation scores are 80% and 54% for our metric and LIC, respectively. The code is available at https://eslambakr.github.io/imagecaptioner2.github.io/.

翻译：大多数预训练学习系统已知存在偏见，这种偏见通常源于数据、模型或两者共同作用。测量和量化偏见及其来源是一项具有挑战性的任务，已在图像描述领域得到广泛研究。尽管已有大量相关研究，但我们观察到现有指标在纳入视觉信号方面缺乏一致性。本文提出一种名为ImageCaptioner$^2$的新型偏见评估指标，用于图像描述。不同于测量模型或数据中的绝对偏见，ImageCaptioner$^2$更关注模型相对于数据偏见引入的额外偏见，即偏见放大。与仅基于生成描述评估图像描述算法的现有方法不同，ImageCaptioner$^2$在测量偏见时将图像纳入考量。此外，我们设计了一种将生成描述的偏见度量转化为基于提示的图像描述的新公式，替代了语言分类器方法。最后，我们在三个不同数据集（MS-COCO描述数据集、Artemis V1和Artemis V2）上的11种图像描述架构中，针对三种受保护属性（性别、种族和情感）应用了ImageCaptioner$^2$指标。为验证其有效性，我们提出了一个名为AnonymousBench的新型人工评估范式，专门用于偏见指标的评估。我们的指标在人类一致性方面显著优于最新偏见指标LIC，相关性得分分别为80%（本方法）和54%（LIC）。代码已开源：https://eslambakr.github.io/imagecaptioner2.github.io/。