Diffusion models achieve state-of-the-art performance but often fail to generate outputs that align with human preferences and intentions, resulting in images with poor aesthetic quality and semantic inconsistencies. Existing alignment methods present a difficult trade-off: fine-tuning approaches suffer from loss of diversity with reward over-optimization, while test-time scaling methods introduce significant computational overhead and tend to under-optimize. To address these limitations, we propose HyperAlign, a novel framework that trains a hypernetwork for efficient and effective test-time alignment. Instead of modifying latent states, HyperAlign dynamically generates low-rank adaptation weights to modulate the diffusion model's generation operators. This allows the denoising trajectory to be adaptively adjusted based on input latents, timesteps and prompts for reward-conditioned alignment. We introduce multiple variants of HyperAlign that differ in how frequently the hypernetwork is applied, balancing between performance and efficiency. Furthermore, we optimize the hypernetwork using a reward score objective regularized with preference data to reduce reward hacking. We evaluate HyperAlign on multiple extended generative paradigms, including Stable Diffusion and FLUX. It significantly outperforms existing fine-tuning and test-time scaling baselines in enhancing semantic consistency and visual appeal.

翻译：扩散模型虽能实现最先进的性能，但生成的输出常与人类偏好和意图不一致，导致图像美学质量不佳和语义不一致。现有对齐方法面临艰难的权衡：微调方法因奖励过优化而丧失多样性，而测试时缩放方法则引入显著计算开销且易优化不足。为克服这些局限，我们提出HyperAlign，一种训练超网络以实现高效有效测试时对齐的新框架。HyperAlign不修改潜在状态，而是动态生成低秩适应权重来调制扩散模型的生成算子。这使得去噪轨迹能根据输入潜在变量、时间步和提示进行自适应调整，以实现奖励条件对齐。我们引入了HyperAlign的多种变体，其差异在于超网络的应用频率，以平衡性能与效率。此外，我们使用偏好数据正则化的奖励得分目标来优化超网络，以减少奖励黑客行为。我们在多种扩展生成范式（包括Stable Diffusion和FLUX）上评估HyperAlign。其在增强语义一致性和视觉吸引力方面显著优于现有的微调和测试时缩放基线方法。