As large language models (LLMs) advance their capabilities, aligning these models with human preferences has become crucial. Preference optimization, which trains models to distinguish between preferred and non-preferred responses based on human feedback, has become a crucial component for aligning LLMs. However, most existing works assume noise-free feedback, which is unrealistic due to the inherent errors and inconsistencies in human judgments. This paper addresses the impact of noisy feedback on preference optimization, providing generalization guarantees under these conditions. In particular, we consider noise models that correspond to common real-world sources of noise, such as mislabeling and uncertainty. Unlike traditional analyses that assume convergence, our work focuses on finite-step preference optimization, offering new insights that are more aligned with practical LLM training. We describe how generalization decays with different types of noise across levels of noise rates based on the preference data distribution and number of samples. Our analysis for noisy preference learning applies to a broad family of preference optimization losses such as DPO, IPO, SLiC, etc. Empirical validation on contemporary LLMs confirms the practical relevance of our findings, offering valuable insights for developing AI systems that align with human preferences.

翻译：随着大语言模型（LLM）能力的不断提升，使其与人类偏好对齐变得至关重要。偏好优化通过基于人类反馈训练模型以区分偏好与非偏好响应，已成为实现LLM对齐的关键技术。然而，现有研究大多假设反馈无噪声，这在实际中难以成立，因为人类判断存在固有的误差与不一致性。本文探讨噪声反馈对偏好优化的影响，并在此条件下提供泛化性保证。具体而言，我们考虑了对应现实常见噪声源的噪声模型，例如标注错误和不确定性。与传统分析假设收敛不同，本研究聚焦有限步偏好优化，提出了更贴近实际LLM训练的新见解。我们基于偏好数据分布和样本数量，阐述了在不同噪声类型及噪声率水平下泛化能力的衰减规律。本噪声偏好学习分析适用于包括DPO、IPO、SLiC等在内的广泛偏好优化损失函数族。在当代LLM上的实证验证证实了研究结论的实际相关性，为开发符合人类偏好的AI系统提供了重要参考。