Unbiased Learning to Rank (ULTR) aims to train unbiased ranking models from biased click logs, by explicitly modeling a generation process for user behavior and fitting click data based on examination hypothesis. Previous research found empirically that the true latent relevance is mostly recoverable through click fitting. However, we demonstrate that this is not always achievable, resulting in a significant reduction in ranking performance. This research investigates the conditions under which relevance can be recovered from click data in the first principle. We initially characterize a ranking model as identifiable if it can recover the true relevance up to a scaling transformation, a criterion sufficient for the pairwise ranking objective. Subsequently, we investigate an equivalent condition for identifiability, articulated as a graph connectivity test problem: the recovery of relevance is feasible if and only if the identifiability graph (IG), derived from the underlying structure of the dataset, is connected. The presence of a disconnected IG may lead to degenerate cases and suboptimal ranking performance. To tackle this challenge, we introduce two methods, namely node intervention and node merging, designed to modify the dataset and restore the connectivity of the IG. Empirical results derived from a simulated dataset and two real-world LTR benchmark datasets not only validate our proposed theory but also demonstrate the effectiveness of our methods in alleviating data bias when the relevance model is unidentifiable.

翻译：无偏学习排序旨在通过显式建模用户行为生成过程并基于检验假说拟合点击数据，从有偏点击日志中训练无偏排序模型。先前研究通过实证发现，真实的潜在相关性大多可通过点击拟合恢复。然而，我们证明这并非总能实现，从而导致排序性能显著下降。本研究从第一性原理出发，探讨了相关性可从点击数据中恢复的条件。我们首先将排序模型定义为可识别的——若其能恢复真实相关性至尺度变换程度，该标准对于成对排序目标已足够。随后，我们研究了可识别性的等价条件，将其表述为图连通性检验问题：仅当从数据集底层结构导出的可识别性图连通时，相关性恢复才可行。不连通的可识别性图可能导致退化情形与次优排序性能。为应对此挑战，我们提出节点干预与节点合并两种方法，通过修改数据集以恢复可识别性图的连通性。基于模拟数据集和两个真实世界LTR基准数据集的实证结果，不仅验证了我们提出的理论，还证明了当相关性模型不可识别时，我们的方法在缓解数据偏差方面的有效性。