A well-known problem when learning from user clicks are inherent biases prevalent in the data, such as position or trust bias. Click models are a common method for extracting information from user clicks, such as document relevance in web search, or to estimate click biases for downstream applications such as counterfactual learning-to-rank, ad placement, or fair ranking. Recent work shows that the current evaluation practices in the community fail to guarantee that a well-performing click model generalizes well to downstream tasks in which the ranking distribution differs from the training distribution, i.e., under covariate shift. In this work, we propose an evaluation metric based on conditional independence testing to detect a lack of robustness to covariate shift in click models. We introduce the concept of debiasedness and a metric for measuring it. We prove that debiasedness is a necessary condition for recovering unbiased and consistent relevance scores and for the invariance of click prediction under covariate shift. In extensive semi-synthetic experiments, we show that our proposed metric helps to predict the downstream performance of click models under covariate shift and is useful in an off-policy model selection setting.

翻译：从用户点击行为中学习时，一个广为人知的问题是数据中普遍存在固有偏差，例如位置偏差或信任偏差。点击模型是从用户点击中提取信息的常用方法，可获取网页搜索中的文档相关性，或为反事实学习排序、广告投放、公平排序等下游应用估算点击偏差。最新研究表明，当前学术界的评估实践无法保证性能良好的点击模型能有效泛化至排序分布异于训练分布（即存在协变量偏移）的下游任务。本研究提出一种基于条件独立性检验的评估指标，用于检测点击模型对协变量偏移的鲁棒性缺失。我们引入"无偏性"概念及其量化指标，并证明无偏性是恢复无偏且一致相关性得分的必要条件，也是点击预测在协变量偏移下保持不变性的前提。通过大量半合成实验验证，本文提出的指标有助于预测点击模型在协变量偏移下的下游性能，并在离策略模型选择场景中具有实用价值。