Deep diffusion models have revolutionized image generation by producing high-quality outputs. However, achieving specific objectives with these models often requires costly adaptations such as fine-tuning, which can be resource-intensive and time-consuming. An alternative approach is inference-time control, which involves optimizing the prompt embeddings to guide the generation process without altering the model weights. We explore prompt-embedding search optimization for the Stable Diffusion XL Turbo model, comparing a gradient-free evolutionary approach, the Separable Covariance Matrix Adaptation Evolution Strategy (sep-CMA-ES), against the widely used gradient-based optimizer Adaptive Moment Estimation (Adam). Candidate images are evaluated by a weighted objective that combines LAION Aesthetic Predictor V2 and CLIPScore, enabling explicit trade-offs between aesthetic quality and prompt-image alignment. On 36 prompts sampled from Parti Prompts (P2) under three weight settings (aesthetics-only, balanced, alignment-only), sep-CMA-ES consistently achieves higher objective values than Adam. We additionally analyze divergence from the unoptimized baseline using cosine similarity and SSIM and report the compute and memory footprints. These results suggest that sep-CMA-ES is an effective inference-time optimizer for prompt-embedding search, improving aesthetics-alignment trade-offs and resource usage without model fine-tuning.

翻译：深度扩散模型通过生成高质量输出彻底变革了图像生成领域。然而，利用这些模型实现特定目标往往需要代价高昂的适配手段（如微调），此类方法资源消耗大且耗时长。另一种方案是推理时控制，即通过优化提示嵌入来引导生成过程，而无需修改模型权重。我们针对Stable Diffusion XL Turbo模型探索提示嵌入搜索优化，将无梯度进化方法——可分离协方差矩阵自适应进化策略（sep-CMA-ES）与广泛使用的基于梯度优化器自适应矩估计（Adam）进行对比。候选图像通过结合LAION美学预测器V2和CLIPScore的加权目标函数进行评估，从而实现美学质量与提示-图像对齐度之间的显式权衡。在从Parti Prompts（P2）中采样的36个提示下，针对三种权重设置（仅美学、平衡、仅对齐），sep-CMA-ES始终比Adam获得更高的目标函数值。我们还利用余弦相似度和SSIM分析偏离未优化基线的差异，并报告计算与内存占用情况。这些结果表明，sep-CMA-ES是一种有效的推理时提示嵌入搜索优化器，可在无需模型微调的情况下改善美学-对齐权衡及资源利用。