Active learning can improve the efficiency of training prediction models by identifying the most informative new labels to acquire. However, non-response to label requests can impact active learning's effectiveness in real-world contexts. We conceptualise this degradation by considering the type of non-response present in the data, demonstrating that biased non-response is particularly detrimental to model performance. We argue that biased non-response is likely in contexts where the labelling process, by nature, relies on user interactions. To mitigate the impact of biased non-response, we propose a cost-based correction to the sampling strategy--the Upper Confidence Bound of the Expected Utility (UCB-EU)--that can, plausibly, be applied to any active learning algorithm. Through experiments, we demonstrate that our method successfully reduces the harm from labelling non-response in many settings. However, we also characterise settings where the non-response bias in the annotations remains detrimental under UCB-EU for specific sampling methods and data generating processes. Finally, we evaluate our method on a real-world dataset from an e-commerce platform. We show that UCB-EU yields substantial performance improvements to conversion models that are trained on clicked impressions. Most generally, this research serves to both better conceptualise the interplay between types of non-response and model improvements via active learning, and to provide a practical, easy-to-implement correction that mitigates model degradation.

翻译：主动学习可通过识别最具信息量的新标签来提升预测模型训练效率。然而，现实场景中标签请求的无回应现象会削弱主动学习的有效性。我们通过考量数据中存在不同类型的无回应问题来概念化这种性能退化，证明有偏无回应会特别损害模型性能。我们论证了在标签过程本质上依赖用户交互的背景下，有偏无回应现象很可能出现。为缓解有偏无回应的影响，我们提出一种基于成本的采样策略修正方法——期望效用的上置信界（UCB-EU），该方法原则上可应用于任意主动学习算法。实验表明，我们的方法能在多种场景下有效降低标签无回应带来的危害。但同时我们也发现，对于特定采样方法和数据生成过程，在UCB-EU框架下标注中的无回应偏差仍会产生负面影响。最后，我们在电子商务平台真实数据集上评估了该方法，证明UCB-EU能显著提升基于点击曝光训练的转化模型性能。总体而言，本研究不仅有助于更深入地概念化无回应类型与主动学习模型改进之间的相互作用，还提供了缓解模型性能退化的实用易行修正方案。