Accurate representation in media is known to improve the well-being of the people who consume it. Generative image models trained on large web-crawled datasets such as LAION are known to produce images with harmful stereotypes and misrepresentations of cultures. We improve inclusive representation in generated images by (1) engaging with communities to collect a culturally representative dataset that we call the Cross-Cultural Understanding Benchmark (CCUB) and (2) proposing a novel Self-Contrastive Fine-Tuning (SCoFT) method that leverages the model's known biases to self-improve. SCoFT is designed to prevent overfitting on small datasets, encode only high-level information from the data, and shift the generated distribution away from misrepresentations encoded in a pretrained model. Our user study conducted on 51 participants from 5 different countries based on their self-selected national cultural affiliation shows that fine-tuning on CCUB consistently generates images with higher cultural relevance and fewer stereotypes when compared to the Stable Diffusion baseline, which is further improved with our SCoFT technique.

翻译：媒体中的准确呈现已被证实能改善受众的福祉。基于大规模网络爬取数据集（如LAION）训练的生成式图像模型，已知会产生包含有害刻板印象和文化误读的图像。我们通过以下方式提升生成图像中的包容性表征：(1) 与社群合作收集具有文化代表性的数据集——跨文化理解基准(CCUB)；(2) 提出新型自对比微调(SCoFT)方法，通过利用模型自身已知偏见实现自我优化。SCoFT旨在防止小数据集过拟合、仅编码数据中的高层级信息，并将生成分布从预训练模型编码的误读表征中偏移。我们基于51名来自5个国家的参与者（依据其自选民族文化归属）开展的用户研究表明，相较于Stable Diffusion基线模型，在CCUB上微调能持续生成文化相关性更高且刻板印象更少的图像，而我们的SCoFT技术进一步提升了这一效果。