Estimating item difficulty through field-testing is often resource-intensive and time-consuming. As such, there is strong motivation to develop methods that can predict item difficulty at scale using only the item content. Large Language Models (LLMs) represent a new frontier for this goal. The present research examines the feasibility of using an LLM to predict item difficulty for K-5 mathematics and reading assessment items (N = 5170). Two estimation approaches were implemented: (a) a direct estimation method that prompted the LLM to assign a single difficulty rating to each item, and (b) a feature-based strategy where the LLM extracted multiple cognitive and linguistic features, which were then used in ensemble tree-based models (random forests and gradient boosting) to predict difficulty. Overall, direct LLM estimates showed moderate to strong correlations with true item difficulties. However, their accuracy varied by grade level, often performing worse for early grades. In contrast, the feature-based method yielded stronger predictive accuracy, with correlations as high as r = 0.87 and lower error estimates compared to both direct LLM predictions and baseline regressors. These findings highlight the promise of LLMs in streamlining item development and reducing reliance on extensive field testing and underscore the importance of structured feature extraction. We provide a seven-step workflow for testing professionals who would want to implement a similar item difficulty estimation approach with their item pool.

翻译：通过实地测试来估计题目难度通常需要大量资源和时间。因此，开发仅基于题目内容即可大规模预测题目难度的方法具有强烈的现实需求。大型语言模型为实现这一目标提供了新的前沿途径。本研究探讨了使用LLM预测K-5年级数学与阅读评估题目难度（N = 5170）的可行性。我们实施了两种估计方法：（a）直接估计法：提示LLM为每道题目分配单一难度评分；（b）基于特征的策略：由LLM提取多项认知与语言特征，随后将其输入基于树的集成模型（随机森林与梯度提升）中进行难度预测。总体而言，LLM直接估计结果与真实题目难度呈现中度至强相关性，但其准确性随年级变化，在低年级表现往往较差。相比之下，基于特征的方法展现出更强的预测准确性，其相关性高达r = 0.87，且误差估计低于直接LLM预测及基线回归模型。这些发现凸显了LLM在简化题目开发流程、减少对大规模实地测试依赖方面的潜力，并强调了结构化特征提取的重要性。我们为测试专业人员提供了七步工作流程，以帮助其在自有题库中实施类似的题目难度估计方法。