Despite the remarkable performance of foundation vision-language models, the shared representation space for text and vision can also encode harmful label associations detrimental to fairness. While prior work has uncovered bias in vision-language models' (VLMs) classification performance across geography, work has been limited along the important axis of harmful label associations due to a lack of rich, labeled data. In this work, we investigate harmful label associations in the recently released Casual Conversations datasets containing more than 70,000 videos. We study bias in the frequency of harmful label associations across self-provided labels for age, gender, apparent skin tone, and physical adornments across several leading VLMs. We find that VLMs are $4-13$x more likely to harmfully classify individuals with darker skin tones. We also find scaling transformer encoder model size leads to higher confidence in harmful predictions. Finally, we find improvements on standard vision tasks across VLMs does not address disparities in harmful label associations.

翻译：尽管基础视觉语言模型表现卓越，但其文本与视觉共享表示空间也可能编码有害标签关联，从而损害公平性。虽然先前研究已揭示视觉语言模型（VLM）在地理维度上的分类性能偏差，但由于缺乏丰富的标注数据，关于有害标签关联这一重要维度的研究仍十分有限。本研究针对近期发布的包含超过7万条视频的Casual Conversations数据集，系统探究了有害标签关联现象。我们分析了多个主流VLM在年龄、性别、表观肤色及身体装饰等自报标签维度上有害标签关联频次的偏差。研究发现，VLM对深肤色个体的有害分类概率是浅肤色个体的4-13倍。此外，缩放Transformer编码器模型规模会导致有害预测的置信度升高。最后我们发现，VLM在标准视觉任务上的性能提升并不能消除有害标签关联中的差异。