We derive and investigate two DPO variants that explicitly model the possibility of declaring a tie in pair-wise comparisons. We replace the Bradley-Terry model in DPO with two well-known modeling extensions, by Rao and Kupper and by Davidson, that assign probability to ties as alternatives to clear preferences. Our experiments in neural machine translation and summarization show that explicitly labeled ties can be added to the datasets for these DPO variants without the degradation in task performance that is observed when the same tied pairs are presented to DPO. We find empirically that the inclusion of ties leads to stronger regularization with respect to the reference policy as measured by KL divergence, and we see this even for DPO in its original form. These findings motivate and enable the inclusion of tied pairs in preference optimization as opposed to simply discarding them.

翻译：我们推导并研究了两种DPO变体，这些变体明确建模了在成对比较中声明平局的可能性。我们使用Rao和Kupper以及Davidson提出的两种著名建模扩展替代了DPO中的Bradley-Terry模型，这些扩展将概率分配给平局作为明确偏好之外的替代选项。我们在神经机器翻译和摘要生成任务中的实验表明，为这些DPO变体添加明确标注的平局数据不会导致任务性能下降，而相同平局数据在原始DPO中则会出现性能退化。通过KL散度测量，我们经验性地发现包含平局数据会相对于参考策略产生更强的正则化效应，即使在原始形式的DPO中也能观察到这一现象。这些发现为在偏好优化中纳入平局数据而非简单丢弃提供了理论依据和实践可行性。