This study explores the robustness of university assessments against the use of Open AI's Generative Pre-Trained Transformer 4 (GPT-4) generated content and evaluates the ability of academic staff to detect its use when supported by the Turnitin Artificial Intelligence (AI) detection tool. The research involved twenty-two GPT-4 generated submissions being created and included in the assessment process to be marked by fifteen different faculty members. The study reveals that although the detection tool identified 91% of the experimental submissions as containing some AI-generated content, the total detected content was only 54.8%. This suggests that the use of adversarial techniques regarding prompt engineering is an effective method in evading AI detection tools and highlights that improvements to AI detection software are needed. Using the Turnitin AI detect tool, faculty reported 54.5% of the experimental submissions to the academic misconduct process, suggesting the need for increased awareness and training into these tools. Genuine submissions received a mean score of 54.4, whereas AI-generated content scored 52.3, indicating the comparable performance of GPT-4 in real-life situations. Recommendations include adjusting assessment strategies to make them more resistant to the use of AI tools, using AI-inclusive assessment where possible, and providing comprehensive training programs for faculty and students. This research contributes to understanding the relationship between AI-generated content and academic assessment, urging further investigation to preserve academic integrity.

翻译：本研究探讨了大学评估对OpenAI生成式预训练变换器4（GPT-4）生成内容的鲁棒性，并评估了学术人员在Turnitin人工智能（AI）检测工具辅助下识别此类内容的能力。研究生成了22份GPT-4生成的实验性答卷，并将其纳入评估流程，由15位不同教职人员评分。结果表明，尽管检测工具识别出91%的实验性答卷包含AI生成内容，但整体检测出的内容比例仅为54.8%。这提示对抗性提示工程策略是规避AI检测工具的有效方法，并突显出改进AI检测软件的必要性。借助Turnitin AI检测工具，教职人员将54.5%的实验性答卷上报至学术不端处理流程，表明需提升对这些工具的认识与培训。真实答卷的平均得分为54.4，而AI生成内容得分为52.3，表明GPT-4在真实情境中的表现相当。建议调整评估策略以增强其对AI工具使用的抵御能力，尽可能采用融入AI的评估方式，并为教职人员和学生提供全面培训项目。本研究有助于理解AI生成内容与学术评估之间的关系，并呼吁进一步研究以维护学术诚信。