Subject-driven generation has garnered significant interest recently due to its ability to personalize text-to-image generation. Typical works focus on learning the new subject's private attributes. However, an important fact has not been taken seriously that a subject is not an isolated new concept but should be a specialization of a certain category in the pre-trained model. This results in the subject failing to comprehensively inherit the attributes in its category, causing poor attribute-related generations. In this paper, motivated by object-oriented programming, we model the subject as a derived class whose base class is its semantic category. This modeling enables the subject to inherit public attributes from its category while learning its private attributes from the user-provided example. Specifically, we propose a plug-and-play method, Subject-Derived regularization (SuDe). It constructs the base-derived class modeling by constraining the subject-driven generated images to semantically belong to the subject's category. Extensive experiments under three baselines and two backbones on various subjects show that our SuDe enables imaginative attribute-related generations while maintaining subject fidelity. Codes will be open sourced soon at FaceChain (https://github.com/modelscope/facechain).
翻译:主体驱动生成因其个性化文本到图像生成的能力而引起了广泛兴趣。典型工作侧重于学习新主体的私有属性。然而,一个重要事实未被充分重视:主体并非孤立的新概念,而应是预训练模型中某个类别的具体化。这导致主体无法全面继承其类别的属性,造成属性相关生成效果较差。本文受面向对象编程启发,将主体建模为其语义类别作为基类的派生类。这种建模使主体在从用户提供的示例中学习私有属性的同时,能够继承其类别的公共属性。具体而言,我们提出了一种即插即用的方法——主体派生正则化(SuDe)。该方法通过约束主体驱动生成的图像在语义上属于该主体的类别,构建了基类-派生类建模。在多种主体上基于三个基线和两个骨干网络的广泛实验表明,我们的SuDe在保持主体保真度的同时,实现了富有想象力的属性相关生成。代码将在FaceChain(https://github.com/modelscope/facechain)上开源。