Aligning large language models to preference data is commonly implemented by assuming a known link function between the distribution of observed preferences and the unobserved rewards (e.g., a logistic link as in Bradley-Terry). If the link is wrong, however, inferred rewards can be biased and policies be misaligned. We study policy alignment to preferences under an unknown and unrestricted link. We consider an $f$-divergence-constrained reward maximization problem and show that realizability of the solution in a policy class implies a semiparametric single-index binary choice model, where a scalar-valued index determined by a policy captures the dependence on demonstrations and the rest of the preference distribution is an unrestricted function thereof. Rather than focus on estimation of identifiable finite-dimensional structural parameters in the index as in econometrics, we focus on policy learning, focusing on error to the optimal policy and allowing unidentifiable and nonparametric indices. We develop a variety of policy learners based on profiling the link function, orthogonalizing the link function, and using link-agnostic bipartite ranking objectives. We analyze these and provide finite-sample policy error bounds that depend on generic functional complexity measures of the index class. We further consider practical implementations using first-order optimization suited to neural networks and batched data. The resulting methods are robust to unknown preference noise distribution and scale, while preserving the direct optimization of policies without explicitly fitting rewards.

翻译：将大型语言模型与偏好数据对齐通常通过假设观测偏好分布与未观测奖励之间存在已知连接函数（例如，Bradley-Terry模型中的逻辑连接函数）来实现。然而，若连接函数设定错误，推断出的奖励可能存在偏差，导致策略失准。本研究探讨在连接函数未知且无约束条件下的策略偏好对齐问题。我们考虑一个$f$散度约束的奖励最大化问题，并证明解在策略类中的可实现性意味着一个半参数单指标二元选择模型：其中由策略决定的标量指标捕捉了对演示样本的依赖关系，而偏好分布的其余部分则是该指标的无约束函数。不同于计量经济学中聚焦于指标内可识别有限维结构参数的估计，我们专注于策略学习，着眼于与最优策略的误差，并允许不可识别及非参数化指标。我们开发了多种基于连接函数剖析、连接函数正交化以及连接函数无关二分排序目标的策略学习器。我们分析了这些方法，并提供了依赖于指标类通用函数复杂度度量的有限样本策略误差界。进一步，我们探讨了适用于神经网络和批量数据的一阶优化实用实现方案。所得方法对未知偏好噪声分布和尺度具有鲁棒性，同时保持了无需显式拟合奖励而直接优化策略的特性。