Quality-sensitive applications of machine learning (ML) require quality assurance (QA) by humans before the predictions of an ML model can be deployed. QA for ML (QA4ML) interfaces require users to view a large amount of data and perform many interactions to correct errors made by the ML model. An optimized user interface (UI) can significantly reduce interaction costs. While UI optimization can be informed by user studies evaluating design options, this approach is not scalable because there are typically numerous small variations that can affect the efficiency of a QA4ML interface. Hence, we propose using simulation to evaluate and aid the optimization of QA4ML interfaces. In particular, we focus on simulating the combined effects of human intelligence in initiating appropriate interaction commands and machine intelligence in providing algorithmic assistance for accelerating QA4ML processes. As QA4ML is usually labor-intensive, we use the simulated task completion time as the metric for UI optimization under different interface and algorithm setups. We demonstrate the usage of this UI design method in several QA4ML applications.

翻译：质量敏感的机器学习应用需要在部署模型预测前进行人工质量保障。机器学习质量保障（QA4ML）界面要求用户查看大量数据并进行多次交互以修正模型错误。优化后的用户界面可显著降低交互成本。虽然通过用户研究评估设计方案可指导界面优化，但该方法缺乏可扩展性——影响QA4ML界面效率的细微变体通常数量众多。因此，我们提出采用仿真方法评估并辅助优化QA4ML界面。具体而言，我们重点模拟人类智能（发起适当交互指令）与机器智能（提供算法辅助加速QA4ML流程）的协同效应。鉴于QA4ML通常属于劳动密集型作业，我们以仿真任务完成时间作为不同界面与算法配置下的界面优化指标。我们在多个QA4ML应用中展示了该界面设计方法的使用案例。