As Large Language Models (LLMs) increasingly automate writing tasks, there is a growing risk of cognitive deskilling where users offload critical thinking to the system. To address this, we introduce Critical Inker, a writing tool designed to scaffold critical reflection during writing through logical analysis and socratic feedback. We present two methods: (1) A Socratic chatbot using questions to help them realize and fix logical errors in their writing and (2) Visual Feedback, which highlights logical errors in the text without dialog. We detail the technical implementation of the system and evaluate its argument extraction and logical validity accuracy. Our evaluation shows a 91.2% argument overlap with ground truth argument annotations and 87% validity accuracy. Finally, we conducted a small-scale pilot and discuss early qualitative results.
翻译:摘要:随着大型语言模型(LLM)越来越多地自动化写作任务,用户将批判性思维外包给系统的认知技能退化的风险日益增加。为解决这一问题,我们提出了Critical Inker——一款旨在通过逻辑分析和苏格拉底式反馈在写作过程中支撑批判性反思的写作工具。我们提供两种方法:(1)一种苏格拉底式聊天机器人,通过提问帮助用户识别并修正写作中的逻辑错误;(2)可视化反馈,在无需对话的情况下突出显示文本中的逻辑错误。我们详细阐述了系统的技术实现,并评估了其论点提取和逻辑有效性准确性。评估结果显示,论点与真实标注的重叠率达到91.2%,有效性准确率为87%。最后,我们开展了一项小规模试点研究,并讨论了初步的定性结果。