Automated short-answer scoring lags other LLM applications. We meta-analyze 890 culminating results across a systematic review of LLM short-answer scoring studies, modeling the traditional effect size of Quadratic Weighted Kappa (QWK) with mixed effects metaregression. We quantitatively illustrate that that the level of difficulty for human experts to perform the task of scoring written work of children has no observed statistical effect on LLM performance. Particularly, we show that some scoring tasks measured as the easiest by human scorers were the hardest for LLMs. Whether by poor implementation by thoughtful researchers or patterns traceable to autoregressive training, on average decoder-only architectures underperform encoders by 0.37--a substantial difference in agreement with humans. Additionally, we measure the contributions of various aspects of LLM technology on successful scoring such as tokenizer vocabulary size, which exhibits diminishing returns--potentially due to undertrained tokens. Findings argue for systems design which better anticipates known statistical shortcomings of autoregressive models. Finally, we provide additional experiments to illustrate wording and tokenization sensitivity and bias elicitation in high-stakes education contexts, where LLMs demonstrate racial discrimination. Code and data for this study are available.

翻译：自动简答题评分技术落后于其他大语言模型应用。我们通过对大语言模型简答题评分研究的系统性综述，对890项最终结果进行元分析，采用混合效应元回归模型计算二次加权卡帕系数的传统效应量。定量研究表明：人类专家评估儿童书面作业的任务难度水平对大语言模型表现无显著统计效应。值得注意的是，某些被人类评分者判定为最简单的评分任务，对大语言模型而言反而最困难。无论是由于深思熟虑的研究者实施不当，还是可追溯至自回归训练模式的固有缺陷，平均而言仅解码器架构模型比编码器模型表现低0.37——这在人类评分一致性方面构成显著差异。此外，我们量化评估了大语言模型技术各维度对评分成功率的影响，例如分词器词汇量呈现收益递减现象——可能源于未充分训练的词元。研究结果论证了系统设计需更好地预判自回归模型的已知统计缺陷。最后，我们通过补充实验揭示了高风险教育场景中存在的措辞敏感性、分词偏差及偏见诱发问题，其中大语言模型表现出种族歧视倾向。本研究的代码与数据已公开。