We present an empirical study of how both experienced tutors and non-tutors judge the correctness of tutor praise responses under different Artificial Intelligence (AI)-assisted interfaces, types of explanation (textual explanations vs. inline highlighting). We first fine-tuned several Large Language Models (LLMs) to produce binary correctness labels and explanations, achieving up to 88% accuracy and 0.92 F1 score with GPT-4. We then let the GPT-4 models assist 95 participants in tutoring decision-making tasks by offering different types of explanations. Our findings show that although human-AI collaboration outperforms humans alone in evaluating tutor responses, it remains less accurate than AI alone. Moreover, we find that non-tutors tend to follow the AI's advice more consistently, which boosts their overall accuracy on the task: especially when the AI is correct. In contrast, experienced tutors often override the AI's correct suggestions and thus miss out on potential gains from the AI's generally high baseline accuracy. Further analysis reveals that explanations in text reasoning will increase over-reliance and reduce underreliance, while inline highlighting does not. Moreover, neither explanation style actually has a significant effect on performance and costs participants more time to complete the task, instead of saving time. Our findings reveal a tension between expertise, explanation design, and efficiency in AI-assisted decision-making, highlighting the need for balanced approaches that foster more effective human-AI collaboration.

翻译：本研究通过实证方法，探讨了经验丰富的辅导教师与非辅导人员在人工智能辅助界面下，如何评判辅导教师表扬性回应的正确性。实验对比了两种解释呈现形式：文本解释与行内高亮。我们首先对多个大语言模型进行微调，使其生成二元正确性标签及解释，其中GPT-4模型取得了最高88%的准确率与0.92的F1分数。随后，我们让GPT-4模型通过提供不同类型的解释，辅助95名参与者完成辅导决策任务。研究发现：虽然人机协作在评估辅导回应方面优于纯人工判断，但其准确性仍低于纯AI判断。此外，非辅导人员倾向于更稳定地遵循AI建议，这提升了他们在任务中的整体准确率——尤其在AI判断正确时更为明显。相比之下，经验丰富的辅导教师常会推翻AI的正确建议，因而未能充分利用AI普遍较高的基线准确率所带来的潜在增益。进一步分析表明，文本推理类解释会加剧过度依赖并降低依赖不足，而行内高亮则无此效应。值得注意的是，两种解释形式均未对任务绩效产生显著影响，反而增加了参与者的任务完成时间而非节省时间。本研究揭示了AI辅助决策中专业知识、解释设计与效率之间的张力，强调需要采取平衡策略以促进更有效的人机协作。