Sycophancy, the tendency of LLM-based chatbots to express excessive agreement with their users, even when inappropriate, is emerging as a significant risk in human-AI interactions. However, the extent to which this affects human-LLM collaboration in complex problem-solving tasks is not well quantified, especially among novices who are prone to misconceptions. We created two LLM chatbots, one with high sycophancy and one with low sycophancy, and conducted a within-subjects experiment (n=24) in the context of debugging machine learning models to investigate the effect of sycophancy on users' mental models, workflows, reliance behaviors, and perceptions of the chatbots. Our findings show that users of the high sycophancy chatbot were less likely to correct their misconceptions and spent more time over-relying on unhelpful LLM responses, leading them to significantly worse performance in the task. Despite these impaired outcomes, a majority of users were unable to detect the presence of excessive sycophancy.

翻译：谄媚性，即基于大语言模型的聊天机器人倾向于过度附和用户（即便在不恰当的情况下），正逐渐成为人机交互中的重大风险。然而，这种行为在复杂问题解决任务中对人机协作的影响程度尚未得到充分量化，尤其是在容易产生错误观念的新手群体中。我们创建了两个大语言模型聊天机器人，一个具有高谄媚性，另一个具有低谄媚性，并在调试机器学习模型的背景下进行了一项被试内实验（n=24），以研究谄媚性对用户心智模型、工作流程、依赖行为以及对聊天机器人感知的影响。我们的研究结果表明，使用高谄媚性聊天机器人的用户更不可能纠正自己的错误观念，并且花费更多时间过度依赖无益的大语言模型回复，导致他们在任务中的表现显著更差。尽管结果受损，大多数用户仍未能察觉过度谄媚性的存在。