The application of Unbiased Learning to Rank (ULTR) is widespread in modern systems for training unbiased ranking models from biased click logs. The key is to explicitly model a generation process for user behavior and fit click data based on examination hypothesis. Previous research found empirically that the true latent relevance can be recovered in most cases as long as the clicks are perfectly fitted. However, we demonstrate that this is not always achievable, resulting in a significant reduction in ranking performance. In this work, we aim to answer if or when the true relevance can be recovered from click data, which is a foundation issue for ULTR field. We first define a ranking model as identifiable if it can recover the true relevance up to a scaling transformation, which is enough for pairwise ranking objective. Then we explore an equivalent condition for identifiability that can be novely expressed as a graph connectivity test problem: if and only if a graph (namely identifiability graph, or IG) constructed on the underlying structure of the dataset is connected, we can guarantee that the relevance can be correctly recovered. When the IG is not connected, there may be bad cases leading to poor ranking performance. To address this issue, we propose two methods, namely node intervention and node merging, to modify the dataset and restore connectivity of the IG. Empirical results obtained on a simulation dataset and two LTR benchmark datasets confirm the validity of our proposed theorems and show the effectiveness of our methods in mitigating data bias when the relevance model is unidentifiable.

翻译：无偏学习排序（ULTR）广泛应用于现代系统中，用于从有偏点击日志训练无偏排序模型。其关键在于显式建模用户行为生成过程，并基于检验假设拟合点击数据。先前研究发现，只要点击数据被完美拟合，潜在的真实相关性在大多数情况下可被恢复。然而，我们证明这并非总能实现，从而导致排序性能显著下降。本文旨在回答一个ULTR领域的基础性问题：能否以及何时从点击数据中恢复真实相关性？我们首先定义排序模型为可辨识的，若其能恢复真实相关性（允许缩放变换），该条件对成对排序目标已足够。随后，我们发现可辨识性的等价条件可创新性地表示为图连通性检验问题：当且仅当基于数据集底层结构构建的图（称为可辨识图，IG）连通时，才能保证相关性被正确恢复。若IG不连通，可能出现导致排序性能低下的不良情形。为解决此问题，我们提出两种方法——节点干预和节点合并——用以修改数据集并恢复IG连通性。在模拟数据集和两个LTR基准数据集上的实验证实了我们所提定理的有效性，并展示了在相关性模型不可辨识时，我们的方法能有效缓解数据偏差。