Title: 聊天GPT和Bard能否生成对齐的评估题目？针对人类表现的可靠性分析 Abstract: 聊天GPT和Bard是基于大型语言模型的AI chatbot，被指定为在不同领域中承诺不同应用的技术。在教育方面，这些AI技术已经测试用于评估和教学的应用。在评估中，AI长期以来已经被用于自动化论文评分和自动化项目生成。这些工具必须具有高可靠性，以在评估中协助或替换人类，即AI得分和人类评分之间的一致性。在本文中，我们通过训练有素的人类来衡量OpenAI ChatGPT和Google Bard LLMs工具对写作提示复杂性的感知和评分的可靠性。作为绩效指标的ICC显示，OpenAI ChatGPT和Google Bard的相互可靠性与人类评分的黄金标准相比很低。 (Can ChatGPT and Bard Generate Aligned Assessment Items? A Reliability Analysis against Human Performance)

翻译：Title: 聊天GPT和Bard能否生成对齐的评估题目？针对人类表现的可靠性分析 Abstract: 聊天GPT和Bard是基于大型语言模型的AI chatbot，被指定为在不同领域中承诺不同应用的技术。在教育方面，这些AI技术已经测试用于评估和教学的应用。在评估中，AI长期以来已经被用于自动化论文评分和自动化项目生成。这些工具必须具有高可靠性，以在评估中协助或替换人类，即AI得分和人类评分之间的一致性。在本文中，我们通过训练有素的人类来衡量OpenAI ChatGPT和Google Bard LLMs工具对写作提示复杂性的感知和评分的可靠性。作为绩效指标的ICC显示，OpenAI ChatGPT和Google Bard的相互可靠性与人类评分的黄金标准相比很低。

Abdolvahab Khademi

ChatGPT and Bard are AI chatbots based on Large Language Models (LLM) that are slated to promise different applications in diverse areas. In education, these AI technologies have been tested for applications in assessment and teaching. In assessment, AI has long been used in automated essay scoring and automated item generation. One psychometric property that these tools must have to assist or replace humans in assessment is high reliability in terms of agreement between AI scores and human raters. In this paper, we measure the reliability of OpenAI ChatGP and Google Bard LLMs tools against experienced and trained humans in perceiving and rating the complexity of writing prompts. Intraclass correlation (ICC) as a performance metric showed that the inter-reliability of both the OpenAI ChatGPT and the Google Bard were low against the gold standard of human ratings.

翻译：