Recent text-to-image generative models have demonstrated an unparalleled ability to generate diverse and creative imagery guided by a target text prompt. While revolutionary, current state-of-the-art diffusion models may still fail in generating images that fully convey the semantics in the given text prompt. We analyze the publicly available Stable Diffusion model and assess the existence of catastrophic neglect, where the model fails to generate one or more of the subjects from the input prompt. Moreover, we find that in some cases the model also fails to correctly bind attributes (e.g., colors) to their corresponding subjects. To help mitigate these failure cases, we introduce the concept of Generative Semantic Nursing (GSN), where we seek to intervene in the generative process on the fly during inference time to improve the faithfulness of the generated images. Using an attention-based formulation of GSN, dubbed Attend-and-Excite, we guide the model to refine the cross-attention units to attend to all subject tokens in the text prompt and strengthen - or excite - their activations, encouraging the model to generate all subjects described in the text prompt. We compare our approach to alternative approaches and demonstrate that it conveys the desired concepts more faithfully across a range of text prompts.

翻译：近期文本到图像生成模型展现了在目标文本提示引导下生成多样化和创造性图像的非凡能力。尽管具有革命性，当前最先进的扩散模型仍可能无法完全传达给定文本提示中的语义信息。我们分析了公开可用的Stable Diffusion模型，并评估了灾难性忽略现象的存在——即模型未能生成输入提示中的一个或多个主体。此外，我们发现某些情况下模型也无法正确地将属性（如颜色）绑定到对应主体。为缓解这些失败案例，我们引入生成式语义护理（GSN）概念，旨在推理过程中动态干预生成进程，以提高生成图像的忠实度。通过基于注意力的GSN公式（称为关注-激发），我们引导模型优化交叉注意力单元，使其关注文本提示中的所有主体标记，并增强（激发）其激活值，从而鼓励模型生成文本描述中的所有主体。我们将本方法与替代方案进行比较，证明其在多种文本提示下能更忠实地传达所需概念。