Improving our understanding of how humans perceive AI teammates is an important foundation for our general understanding of human-AI teams. Extending relevant work from cognitive science, we propose a framework based on item response theory for modeling these perceptions. We apply this framework to real-world experiments, in which each participant works alongside another person or an AI agent in a question-answering setting, repeatedly assessing their teammate's performance. Using this experimental data, we demonstrate the use of our framework for testing research questions about people's perceptions of both AI agents and other people. We contrast mental models of AI teammates with those of human teammates as we characterize the dimensionality of these mental models, their development over time, and the influence of the participants' own self-perception. Our results indicate that people expect AI agents' performance to be significantly better on average than the performance of other humans, with less variation across different types of problems. We conclude with a discussion of the implications of these findings for human-AI interaction.

翻译：提升对人类如何感知AI队友的理解，是我们理解人-AI团队运作的重要基础。借鉴认知科学的相关研究，我们提出了一种基于项目反应理论的框架来建模这些感知。我们将该框架应用于真实世界的实验，在实验中，每位参与者与另一个人或AI代理者在问答场景中协作，并反复评估其队友的表现。利用实验数据，我们展示了该框架在检验关于人们对AI代理者及其他人类感知的研究问题中的用途。通过对比AI队友与人类队友的心理模型，我们刻画了这些心理模型的维度、随时间演变的过程，以及参与者自身自我认知的影响。结果表明，人们通常认为AI代理者的平均表现显著优于其他人类，且在不同类型问题上的表现差异更小。最后，我们讨论了这些发现对人-AI交互的启示。