When designing Machine Learning (ML) enabled solutions, designers often need to simulate ML behavior through the Wizard of Oz (WoZ) approach to test the user experience before the ML model is available. Although reproducing ML errors is essential for having a good representation, they are rarely considered. We introduce Wizard of Errors (WoE), a tool for conducting WoZ studies on ML-enabled solutions that allows simulating ML errors during user experience assessment. We explored how this system can be used to simulate the behavior of a computer vision model. We tested WoE with design students to determine the importance of considering ML errors in design, the relevance of using descriptive error types instead of confusion matrix, and the suitability of manual error control in WoZ studies. Our work identifies several challenges, which prevent realistic error representation by designers in such studies. We discuss the implications of these findings for design.

翻译：设计机器学习驱动的解决方案时，设计师通常需通过"巫术师"方法模拟机器学习行为，以便在模型可用前测试用户体验。尽管复现机器学习错误对获得良好表征至关重要，但它们却鲜少被考虑。我们引入"错误巫师"（WoE），这是一种用于对机器学习解决方案进行巫术师研究的工具，可在用户体验评估期间模拟机器学习错误。我们探究了该系统如何用于模拟计算机视觉模型的行为。我们邀请设计专业学生测试WoE，以确定在设计中考量机器学习错误的重要性、使用描述性错误类型替代混淆矩阵的相关性，以及在巫术师研究中手动控制错误的适用性。我们的研究识别出若干挑战，这些挑战阻碍了设计师在此类研究中实现逼真的错误表征。我们讨论了这些发现对设计实践的启示。