Preference learning from human feedback has the ability to align generative models with the needs of end-users. Human feedback is costly and time-consuming to obtain, which creates demand for data-efficient query selection methods. This work presents a novel approach that leverages optimal experimental design to ask humans the most informative preference queries, from which we can elucidate the latent reward function modeling user preferences efficiently. We formulate the problem of preference query selection as the one that maximizes the information about the underlying latent preference model. We show that this problem has a convex optimization formulation, and introduce a statistically and computationally efficient algorithm ED-PBRL that is supported by theoretical guarantees and can efficiently construct structured queries such as images or text. We empirically present the proposed framework by personalizing a text-to-image generative model to user-specific styles, showing that it requires less preference queries compared to random query selection.

翻译：从人类反馈中学习偏好能够使生成模型与终端用户的需求保持一致。获取人类反馈成本高昂且耗时，这催生了对数据高效查询选择方法的需求。本研究提出一种创新方法，利用最优实验设计向人类提出最具信息量的偏好查询，从而高效揭示建模用户偏好的潜在奖励函数。我们将偏好查询选择问题形式化为最大化潜在偏好模型信息获取的优化问题。研究表明该问题具有凸优化形式，并提出具备理论保证的统计与计算高效算法ED-PBRL，该算法能有效构建图像或文本等结构化查询。通过将文本到图像生成模型个性化适配至用户特定风格的实证研究，我们证明该方法相较于随机查询选择能显著减少偏好查询需求。