Binary choices, as often used for reinforcement learning from human feedback (RLHF), convey only the direction of a preference. A person may choose apples over oranges and bananas over grapes, but which preference is stronger? Strength is crucial for decision-making under uncertainty and generalization of preference models, but hard to measure reliably. Metadata such as response times and inter-annotator agreement can serve as proxies for strength, but are often noisy and confounded. We propose ResponseRank to address the challenge of learning from noisy strength signals. Our method uses relative differences in proxy signals to rank responses to pairwise comparisons by their inferred preference strength. To control for systemic variation, we compare signals only locally within carefully constructed strata. This enables robust learning of utility differences consistent with strength-derived rankings while making minimal assumptions about the strength signal. Our contributions are threefold: (1) ResponseRank, a novel method that robustly learns preference strength by leveraging locally valid relative strength signals; (2) empirical evidence of improved sample efficiency and robustness across diverse tasks: synthetic preference learning (with simulated response times), language modeling (with annotator agreement), and RL control tasks (with simulated episode returns); and (3) the Pearson Distance Correlation (PDC), a novel metric that isolates cardinal utility learning from ordinal accuracy.

翻译：二元选择（常用于人类反馈强化学习RLHF）仅传达偏好的方向。一个人可能选择苹果而非橙子，选择香蕉而非葡萄，但哪种偏好更强？强度对于不确定性下的决策和偏好模型的泛化至关重要，却难以可靠测量。响应时间和标注者间一致性等元数据可作为强度的代理指标，但通常存在噪声且易受混杂因素影响。本文提出ResponseRank以应对从噪声强度信号中学习的挑战。该方法利用代理信号的相对差异，通过推断的偏好强度对成对比较的响应进行排序。为控制系统性变异，我们仅在精心构建的分层内进行局部信号比较。这使得在最小化强度信号假设的前提下，能够稳健地学习与强度排序一致的效用差异。我们的贡献包括三方面：（1）ResponseRank——一种通过利用局部有效相对强度信号来稳健学习偏好强度的新方法；（2）在多样化任务中验证样本效率与鲁棒性提升的实验证据：合成偏好学习（模拟响应时间）、语言建模（标注者一致性）和强化学习控制任务（模拟回合回报）；（3）皮尔逊距离相关性（PDC）——一种将基数效用学习与序数准确度分离的新评估指标。