A well-known problem when learning from user clicks are inherent biases prevalent in the data, such as position or trust bias. Click models are a common method for extracting information from user clicks, such as document relevance in web search, or to estimate click biases for downstream applications such as counterfactual learning-to-rank, ad placement, or fair ranking. Recent work shows that the current evaluation practices in the community fail to guarantee that a well-performing click model generalizes well to downstream tasks in which the ranking distribution differs from the training distribution, i.e., under covariate shift. In this work, we propose an evaluation metric based on conditional independence testing to detect a lack of robustness to covariate shift in click models. We introduce the concept of debiasedness and a metric for measuring it. We prove that debiasedness is a necessary condition for recovering unbiased and consistent relevance scores and for the invariance of click prediction under covariate shift. In extensive semi-synthetic experiments, we show that our proposed metric helps to predict the downstream performance of click models under covariate shift and is useful in an off-policy model selection setting.

翻译：从用户点击中学习时，一个众所周知的问题是数据中普遍存在的固有偏差，例如位置偏差或信任偏差。点击模型是一种从用户点击中提取信息的常见方法，例如提取网页搜索中的文档相关性，或估计点击偏差以用于下游应用，如反事实学习排序、广告投放或公平排序。最新研究表明，当前领域内的评估实践无法保证一个性能良好的点击模型能够很好地泛化到排名分布与训练分布不同的下游任务中，即在协变量偏移下表现良好。在这项工作中，我们提出了一种基于条件独立性检验的评估指标，用于检测点击模型在协变量偏移下的鲁棒性不足。我们引入了无偏性这一概念及其度量指标。我们证明了无偏性是恢复无偏且一致的相关性分数以及点击预测在协变量偏移下保持不变性的必要条件。在大量的半合成实验中，我们表明所提出的指标有助于预测点击模型在协变量偏移下的下游性能，并且在离线策略模型选择场景中具有实用价值。