The development and deployment of systems using supervised machine learning (ML) remain challenging: mainly due to the limited reliability of prediction models and the lack of knowledge on how to effectively integrate human intelligence into automated decision-making. Humans involvement in the ML process is a promising and powerful paradigm to overcome the limitations of pure automated predictions and improve the applicability of ML in practice. We compile a catalog of design patterns to guide developers select and implement suitable human-in-the-loop (HiL) solutions. Our catalog takes into consideration key requirements as the cost of human involvement and model retraining. It includes four training patterns, four deployment patterns, and two orthogonal cooperation patterns.

翻译：基于监督式机器学习（ML）的系统开发与部署仍面临诸多挑战：主要源于预测模型的可靠性有限，以及缺乏如何有效将人类智能整合到自动化决策中的知识。人类参与机器学习过程是一种有前景且强大的范式，可克服纯自动化预测的局限，提升ML在实践中的适用性。我们整理了一份设计模式目录，用于指导开发者选择并实施合适的人机协同（HiL）解决方案。该目录综合考虑了人类参与成本与模型再训练等关键需求，包含四种训练模式、四种部署模式及两种正交协作模式。