Recent text-to-image generative models have demonstrated an unparalleled ability to generate diverse and creative imagery guided by a target text prompt. While revolutionary, current state-of-the-art diffusion models may still fail in generating images that fully convey the semantics in the given text prompt. We analyze the publicly available Stable Diffusion model and assess the existence of catastrophic neglect, where the model fails to generate one or more of the subjects from the input prompt. Moreover, we find that in some cases the model also fails to correctly bind attributes (e.g., colors) to their corresponding subjects. To help mitigate these failure cases, we introduce the concept of Generative Semantic Nursing (GSN), where we seek to intervene in the generative process on the fly during inference time to improve the faithfulness of the generated images. Using an attention-based formulation of GSN, dubbed Attend-and-Excite, we guide the model to refine the cross-attention units to attend to all subject tokens in the text prompt and strengthen - or excite - their activations, encouraging the model to generate all subjects described in the text prompt. We compare our approach to alternative approaches and demonstrate that it conveys the desired concepts more faithfully across a range of text prompts.

翻译：近期文本到图像生成模型展现出无与伦比的能力，能够根据目标文本提示生成多样且富有创意的图像。尽管具有革命性意义，当前最先进的扩散模型在生成完全传达给定文本提示语义的图像时仍可能失败。我们分析了公开可用的Stable Diffusion模型，评估了灾难性忽略现象的存在，即模型未能生成输入提示中的一个或多个主体。此外，我们发现某些情况下模型还无法正确地将属性（如颜色）绑定到其对应主体。为缓解这些失败案例，我们引入生成性语义护理（GSN）概念，旨在推理阶段动态干预生成过程，以提升生成图像的保真度。基于注意力机制的GSN（称为注意力与激发）引导模型精炼交叉注意力单元，使其关注文本提示中的所有主体令牌，并增强或激发其激活强度，从而促使模型生成文本提示中描述的所有主体。我们将本方法与替代方案进行比较，证明其在广泛文本提示中更忠实地传达所需概念。