While Diffusion Models (DM) exhibit remarkable performance across various image generative tasks, they nonetheless reflect the inherent bias presented in the training set. As DMs are now widely used in real-world applications, these biases could perpetuate a distorted worldview and hinder opportunities for minority groups. Existing methods on debiasing DMs usually requires model retraining with a human-crafted reference dataset or additional classifiers, which suffer from two major limitations: (1) collecting reference datasets causes expensive annotation cost; (2) the debiasing performance is heavily constrained by the quality of the reference dataset or the additional classifier. To address the above limitations, we propose FairGen, a plug-and-play method that learns attribute latent directions in a self-discovering manner, thus eliminating the reliance on such reference dataset. Specifically, FairGen consists of two parts: a set of attribute adapters and a distribution indicator. Each adapter in the set aims to learn an attribute latent direction, and is optimized via noise composition through a self-discovering process. Then, the distribution indicator is multiplied by the set of adapters to guide the generation process towards the prescribed distribution. Our method enables debiasing multiple attributes in DMs simultaneously, while remaining lightweight and easily integrable with other DMs, eliminating the need for retraining. Extensive experiments on debiasing gender, racial, and their intersectional biases show that our method outperforms previous SOTA by a large margin.

翻译：尽管扩散模型（DM）在各种图像生成任务中展现出卓越性能，它们仍会反映训练数据中存在的固有偏见。随着扩散模型在现实应用中的广泛使用，这些偏见可能延续扭曲的世界观并阻碍少数群体的机会。现有的扩散模型去偏方法通常需要基于人工构建的参考数据集或额外分类器进行模型重训练，这些方法存在两大局限：（1）收集参考数据集会产生高昂的标注成本；（2）去偏效果严重受限于参考数据集或额外分类器的质量。为克服上述局限，我们提出FairGen——一种即插即用的方法，通过自发现方式学习属性潜在方向，从而消除对此类参考数据集的依赖。具体而言，FairGen包含两个部分：一组属性适配器和一个分布指示器。该集合中的每个适配器旨在学习一个属性潜在方向，并通过自发现过程中的噪声组合进行优化。随后，分布指示器与适配器集合相乘，引导生成过程朝向预设分布。我们的方法能够同时消除扩散模型中的多重属性偏见，同时保持轻量化并易于与其他扩散模型集成，无需重新训练。在消除性别、种族及其交叉偏见的广泛实验中，我们的方法以显著优势超越了先前的最先进技术。