Integrating human expertise into machine learning systems often reduces the role of experts to labeling oracles, a paradigm that limits the amount of information exchanged and fails to capture the nuances of human judgment. We address this challenge by developing a human-in-the-loop framework to learn binary classifiers with rich query types, consisting of item ranking and exemplar selection. We first introduce probabilistic human response models for these rich queries motivated by the relationship experimentally observed between the perceived implicit score of an item and its distance to the unknown classifier. Using these models, we then design active learning algorithms that leverage the rich queries to increase the information gained per interaction. We provide theoretical bounds on sample complexity and develop a tractable and computationally efficient variational approximation. Through experiments with simulated annotators derived from crowdsourced word-sentiment and image-aesthetic datasets, we demonstrate significant reductions on sample complexity. We further extend active learning strategies to select queries that maximize information rate, explicitly balancing informational value against annotation cost. This algorithm in the word sentiment classification task reduces learning time by more than 57\% compared to traditional label-only active learning.

翻译：将人类专业知识融入机器学习系统时，专家通常被简化为标注工具，这种范式限制了信息交换的深度，且难以捕捉人类判断的细微差别。为应对这一挑战，我们开发了一种人机协同学习框架，通过包含项目排序和范例选择的丰富查询类型来学习二元分类器。我们首先基于实验观察到的项目感知隐式得分与未知分类器距离之间的关系，为这些丰富查询构建了概率化的人类响应模型。随后，我们利用这些模型设计了主动学习算法，通过丰富查询提升单次交互的信息获取效率。我们提供了样本复杂度的理论界，并开发了可处理且计算高效的变分近似方法。通过基于众包词语情感与图像美学数据集构建的模拟标注者进行实验，我们证明了该方法能显著降低样本复杂度。我们进一步扩展了主动学习策略，通过选择能最大化信息率的查询来显式平衡信息价值与标注成本。在词语情感分类任务中，该算法相较于传统仅使用标签的主动学习方法，将学习时间缩短了57%以上。