Information systems increasingly leverage artificial intelligence (AI) and machine learning (ML) to generate value from vast amounts of data. However, ML models are imperfect and can generate incorrect classifications. Hence, human-in-the-loop (HITL) extensions to ML models add a human review for instances that are difficult to classify. This study argues that continuously relying on human experts to handle difficult model classifications leads to a strong increase in human effort, which strains limited resources. To address this issue, we propose a hybrid system that creates artificial experts that learn to classify data instances from unknown classes previously reviewed by human experts. Our hybrid system assesses which artificial expert is suitable for classifying an instance from an unknown class and automatically assigns it. Over time, this reduces human effort and increases the efficiency of the system. Our experiments demonstrate that our approach outperforms traditional HITL systems for several benchmarks on image classification.

翻译：信息系统日益依赖人工智能（AI）和机器学习（ML）从海量数据中创造价值。然而，ML模型并非完美无缺，可能产生错误分类。因此，ML模型的人机协同（HITL）扩展机制引入人工审核环节，以处理难以分类的实例。本研究指出，持续依赖人类专家处理模型难以分类的问题将导致人力投入大幅增加，对有限资源造成压力。为解决此问题，我们提出一种混合系统，该系统能够创建人工专家，使其学习对人类专家先前审核过的未知类别数据实例进行分类。我们的混合系统可评估哪个专家适合对未知类别的实例进行分类，并自动分配任务。随着时间的推移，这种机制降低了人力投入并提升了系统效率。实验结果表明，在多个图像分类基准测试中，我们的方法优于传统HITL系统。