The use of machine learning (ML) models to assess and score textual data has become increasingly pervasive in an array of contexts including natural language processing, information retrieval, search and recommendation, and credibility assessment of online content. A significant disruption at the intersection of ML and text are text-generating large-language models such as generative pre-trained transformers (GPTs). We empirically assess the differences in how ML-based scoring models trained on human content assess the quality of content generated by humans versus GPTs. To do so, we propose an analysis framework that encompasses essay scoring ML-models, human and ML-generated essays, and a statistical model that parsimoniously considers the impact of type of respondent, prompt genre, and the ML model used for assessment model. A rich testbed is utilized that encompasses 18,460 human-generated and GPT-based essays. Results of our benchmark analysis reveal that transformer pretrained language models (PLMs) more accurately score human essay quality as compared to CNN/RNN and feature-based ML methods. Interestingly, we find that the transformer PLMs tend to score GPT-generated text 10-15\% higher on average, relative to human-authored documents. Conversely, traditional deep learning and feature-based ML models score human text considerably higher. Further analysis reveals that although the transformer PLMs are exclusively fine-tuned on human text, they more prominently attend to certain tokens appearing only in GPT-generated text, possibly due to familiarity/overlap in pre-training. Our framework and results have implications for text classification settings where automated scoring of text is likely to be disrupted by generative AI.

翻译：使用机器学习（ML）模型评估和评分文本数据在自然语言处理、信息检索、搜索与推荐以及在线内容可信度评估等多个领域日益普及。ML与文本交叉领域的一项重要变革是文本生成式大型语言模型，例如生成式预训练转换器（GPT）。本文实证评估了基于人类内容训练的ML评分模型在评估人类与GPT生成内容质量方面的差异。为此，我们提出了一种分析框架，该框架整合了论文评分ML模型、人类与ML生成的论文，以及一个简约考量受访者类型、提示体裁和评估模型所用ML模型影响的统计模型。我们利用了一个包含18,460篇人类与GPT生成论文的丰富测试平台。基准分析结果表明，与CNN/RNN及基于特征的ML方法相比，转换器预训练语言模型（PLM）能更准确地评估人类论文质量。有趣的是，我们发现转换器PLM对GPT生成文本的评分平均比人类撰写文本高出10-15%。相反，传统深度学习与基于特征的ML模型则对人类文本的评分显著更高。进一步分析揭示，尽管转换器PLM仅针对人类文本进行微调，但它们更显著地关注仅出现在GPT生成文本中的特定标记，这可能源于预训练中的熟悉度/重叠。我们的框架与结果对文本分类场景具有启示意义，在这些场景中，文本的自动化评估很可能受到生成式AI的冲击。