Information systems increasingly leverage artificial intelligence (AI) and machine learning (ML) to generate value from vast amounts of data. However, ML models are imperfect and can generate incorrect classifications. Hence, human-in-the-loop (HITL) extensions to ML models add a human review for instances that are difficult to classify. This study argues that continuously relying on human experts to handle difficult model classifications leads to a strong increase in human effort, which strains limited resources. To address this issue, we propose a hybrid system that creates artificial experts that learn to classify data instances from unknown classes previously reviewed by human experts. Our hybrid system assesses which artificial expert is suitable for classifying an instance from an unknown class and automatically assigns it. Over time, this reduces human effort and increases the efficiency of the system. Our experiments demonstrate that our approach outperforms traditional HITL systems for several benchmarks on image classification.

翻译：信息系统日益利用人工智能和机器学习从海量数据中创造价值。然而，机器学习模型并不完美，可能产生错误分类。因此，机器学习模型的人机协同扩展引入了对难以分类实例的人工审核。本研究认为，持续依赖人类专家处理困难的模型分类会导致人力投入大幅增加，从而消耗有限的资源。为解决这一问题，我们提出一种混合系统，该系统创建了能够学习分类先前由人类专家审核过的未知类别数据实例的人工专家。我们的混合系统会评估哪个人工专家适合对某个未知类别的实例进行分类，并自动进行分配。随着时间的推移，这减少了人力投入，提高了系统效率。实验表明，在图像分类的多个基准测试中，我们的方法优于传统的人机协同系统。