As AI models evolve, understanding the influence of underlying models on user experience and performance in AI-infused systems becomes critical, particularly while transitioning between different model versions. We studied the influence of model change by conducting two complementary studies in the context of AI-based facial recognition for historical person identification tasks. First, we ran an online experiment where crowd workers interacted with two different facial recognition models: an older version and a recently updated, developer-certified more accurate model. Second, we studied a real-world deployment of these models on a popular historical photo platform through a diary study with 10 users. Our findings sheds light on models affecting human-AI team performance, users' abilities to differentiate between different models, the folk theories they develop, and how these theories influence their preferences. Drawing from these insights, we discuss design implications for updating models in AI-infused systems.

翻译：随着AI模型的持续演进，理解底层模型对AI赋能系统中用户体验与系统性能的影响变得至关重要，尤其是在不同模型版本间切换的过程中。我们以基于AI的面部识别技术用于历史人物辨识任务为背景，通过两项互补性研究探究模型变更带来的影响。首先开展在线实验，让众包工作者分别与面部识别模型的两代版本交互——旧版与经开发者认证精度更高的近期更新版。其次通过一项针对10名用户的日志研究，对上述模型在某热门历史照片平台的实际部署情况进行考察。研究结果揭示了模型对人类-AI协同表现的影响、用户区分不同模型的能力、用户形成的素人理论，以及这些理论如何影响其偏好。基于这些发现，我们探讨了AI赋能系统中模型更新的设计启示。