The rapid adoption of text-to-image diffusion models in society underscores an urgent need to address their biases. Without interventions, these biases could propagate a skewed worldview and restrict opportunities for minority groups. In this work, we frame fairness as a distributional alignment problem. Our solution consists of two main technical contributions: (1) a distributional alignment loss that steers specific characteristics of the generated images towards a user-defined target distribution, and (2) adjusted direct finetuning of diffusion model's sampling process (adjusted DFT), which leverages an adjusted gradient to directly optimize losses defined on the generated images. Empirically, our method markedly reduces gender, racial, and their intersectional biases for occupational prompts. Gender bias is significantly reduced even when finetuning just five soft tokens. Crucially, our method supports diverse perspectives of fairness beyond absolute equality, which is demonstrated by controlling age to a $75\%$ young and $25\%$ old distribution while simultaneously debiasing gender and race. Finally, our method is scalable: it can debias multiple concepts at once by simply including these prompts in the finetuning data. We share code and various fair diffusion model adaptors at https://sail-sg.github.io/finetune-fair-diffusion/.

翻译：文本到图像扩散模型在社会中的快速普及凸显了解决其偏见的紧迫性。若不加以干预，这些偏见可能传播扭曲的世界观并限制少数群体的机会。本文将公平性定义为分布对齐问题，并提出包含两项主要技术贡献的解决方案：(1) 一种分布对齐损失函数，能将生成图像的特定特征引导至用户定义的目标分布；(2) 针对扩散模型采样过程的调整后直接微调（调整后DFT），通过利用调整后的梯度直接优化定义在生成图像上的损失函数。实验表明，我们的方法能显著降低职业提示中的性别、种族及其交叉偏见。即使仅微调五个软标记，性别偏见也能大幅减少。最重要的是，我们的方法支持超越绝对平等的多元公平视角——通过将年龄分布控制为$75\%$年轻与$25\%$年老，同时消除性别与种族偏见，证实了这一点。最后，本方法具有可扩展性：仅需在微调数据中包含相关提示，即可同时消除多重概念的偏见。我们在https://sail-sg.github.io/finetune-fair-diffusion/ 共享代码及多种公平扩散模型适配器。