The study explores the capabilities of OpenAI's ChatGPT in solving different types of physics problems. ChatGPT (with GPT-4) was queried to solve a total of 40 problems from a college-level engineering physics course. These problems ranged from well-specified problems, where all data required for solving the problem was provided, to under-specified, real-world problems where not all necessary data were given. Our findings show that ChatGPT could successfully solve 62.5\% of the well-specified problems, but its accuracy drops to 8.3\% for under-specified problems. Analysis of the model's incorrect solutions revealed three distinct failure modes: 1) failure to construct accurate models of the physical world, 2) failure to make reasonable assumptions about missing data, and 3) calculation errors. The study offers implications for how to leverage LLM-augmented instructional materials to enhance STEM education. The insights also contribute to the broader discourse on AI's strengths and limitations, serving both educators aiming to leverage the technology and researchers investigating human-AI collaboration frameworks for problem-solving and decision-making.

翻译：本研究探讨了OpenAI的ChatGPT在解决不同类型物理问题方面的能力。我们利用ChatGPT（基于GPT-4）处理了大学工程物理课程中的40个问题，这些问题涵盖从所有必要数据均已提供的明确指定型问题，到未提供全部数据的欠指定型真实世界问题。研究结果显示，ChatGPT能成功解决62.5%的明确指定型问题，但在欠指定型问题中准确率降至8.3%。通过分析模型错误解答，我们识别出三种典型失效模式：1) 未能构建准确的物理世界模型；2) 未能对缺失数据做出合理假设；3) 计算错误。本研究为如何利用大语言模型增强型教学材料提升STEM教育提供了启示，同时也有助于深化关于人工智能优缺点的广泛讨论，既服务于希望利用该技术的教育工作者，也为探究人机协作框架以解决问题和决策的研究者提供参考。