The recent advancements in Generative AI have significantly advanced the field of text-to-image generation. The state-of-the-art text-to-image model, Stable Diffusion, is now capable of synthesizing high-quality images with a strong sense of aesthetics. Crafting text prompts that align with the model's interpretation and the user's intent thus becomes crucial. However, prompting remains challenging for novice users due to the complexity of the stable diffusion model and the non-trivial efforts required for iteratively editing and refining the text prompts. To address these challenges, we propose PromptCharm, a mixed-initiative system that facilitates text-to-image creation through multi-modal prompt engineering and refinement. To assist novice users in prompting, PromptCharm first automatically refines and optimizes the user's initial prompt. Furthermore, PromptCharm supports the user in exploring and selecting different image styles within a large database. To assist users in effectively refining their prompts and images, PromptCharm renders model explanations by visualizing the model's attention values. If the user notices any unsatisfactory areas in the generated images, they can further refine the images through model attention adjustment or image inpainting within the rich feedback loop of PromptCharm. To evaluate the effectiveness and usability of PromptCharm, we conducted a controlled user study with 12 participants and an exploratory user study with another 12 participants. These two studies show that participants using PromptCharm were able to create images with higher quality and better aligned with the user's expectations compared with using two variants of PromptCharm that lacked interaction or visualization support.

翻译：生成式AI的最新进展显著推动了文本到图像生成领域的发展。当前最先进的文本到图像模型Stable Diffusion能够合成具有强烈美学感知的高质量图像。因此，构建与模型解释和用户意图相一致的文本提示变得至关重要。然而，由于稳定扩散模型的复杂性以及迭代编辑和优化文本提示所需的非 trivial 努力，提示工程对新手用户而言仍充满挑战。为解决这些问题，我们提出了PromptCharm，一种通过多模态提示工程与优化辅助文本到图像生成的混合主动系统。为了帮助新手用户提示，PromptCharm首先自动优化和改进用户的初始提示。此外，PromptCharm支持用户在大数据库中探索和选择不同图像风格。为协助用户有效优化提示和图像，PromptCharm通过可视化模型的注意力值来呈现模型解释。如果用户注意到生成图像中存在任何不满意区域，他们可以在PromptCharm的丰富反馈循环中通过模型注意力调整或图像修复进一步优化图像。为评估PromptCharm的有效性和可用性，我们开展了一项包含12名参与者的受控用户研究和另一项包含12名参与者的探索性用户研究。两项研究表明，与使用缺乏交互或可视化支持的两种PromptCharm变体相比，使用PromptCharm的参与者能够生成更高质量且更符合用户预期的图像。