Generative models, such as large language models and text-to-image diffusion models, are increasingly used to create visual designs like user interfaces (UIs) and presentation slides. Finetuning and benchmarking these generative models have often relied on datasets of human-annotated design preferences. Yet, due to the subjective and highly personalized nature of visual design, preference varies widely among individuals. In this paper, we study this problem by introducing DesignPref, a dataset of 12k pairwise comparisons of UI design generation annotated by 20 professional designers with multi-level preference ratings. We found that among trained designers, substantial levels of disagreement exist (Krippendorff's alpha = 0.25 for binary preferences). Natural language rationales provided by these designers indicate that disagreements stem from differing perceptions of various design aspect importance and individual preferences. With DesignPref, we demonstrate that traditional majority-voting methods for training aggregated judge models often do not accurately reflect individual preferences. To address this challenge, we investigate multiple personalization strategies, particularly fine-tuning or incorporating designer-specific annotations into RAG pipelines. Our results show that personalized models consistently outperform aggregated baseline models in predicting individual designers' preferences, even when using 20 times fewer examples. Our work provides the first dataset to study personalized visual design evaluation and support future research into modeling individual design taste.

翻译：生成模型，如大型语言模型和文本到图像扩散模型，正日益被用于创建用户界面（UI）和演示文稿幻灯片等视觉设计。这些生成模型的微调和基准测试通常依赖于人类标注的设计偏好数据集。然而，由于视觉设计的主观性和高度个性化特性，偏好在不同个体之间存在广泛差异。本文通过引入DesignPref来研究这一问题，这是一个包含20位专业设计师标注的、具有多级偏好评级的12k个UI设计生成成对比较的数据集。我们发现，在受过训练的设计师中，存在显著的分歧水平（二元偏好的Krippendorff's alpha = 0.25）。这些设计师提供的自然语言理由表明，分歧源于对不同设计方面重要性的不同认知以及个人偏好。利用DesignPref，我们证明传统多数投票方法训练聚合评判模型往往无法准确反映个人偏好。为应对这一挑战，我们研究了多种个性化策略，特别是微调或将设计师特定标注纳入RAG管道。我们的结果表明，个性化模型在预测个体设计师偏好方面始终优于聚合基线模型，即使使用少20倍的示例。我们的工作提供了首个研究个性化视觉设计评估的数据集，并支持未来对个体设计品味建模的研究。