In testing industry, precise item categorization is pivotal to align exam questions with the designated content domains outlined in the assessment blueprint. Traditional methods either entail manual classification, which is laborious and error-prone, or utilize machine learning requiring extensive training data, often leading to model underfit or overfit issues. This study unveils a novel approach employing the zero-shot and few-shot Generative Pretrained Transformer (GPT) classifier for hierarchical item categorization, minimizing the necessity for training data, and instead, leveraging human-like language descriptions to define categories. Through a structured python dictionary, the hierarchical nature of examination blueprints is navigated seamlessly, allowing for a tiered classification of items across multiple levels. An initial simulation with artificial data demonstrates the efficacy of this method, achieving an average accuracy of 92.91% measured by the F1 score. This method was further applied to real exam items from the 2022 In-Training Examination (ITE) conducted by the American Board of Family Medicine (ABFM), reclassifying 200 items according to a newly formulated blueprint swiftly in 15 minutes, a task that traditionally could span several days among editors and physicians. This innovative approach not only drastically cuts down classification time but also ensures a consistent, principle-driven categorization, minimizing human biases and discrepancies. The ability to refine classifications by adjusting definitions adds to its robustness and sustainability.

翻译：在测试行业中，精确的试题分类对于将考试题目与评估蓝图中所指定的内容领域对齐至关重要。传统方法要么依赖人工分类（耗时且易出错），要么采用需要大量训练数据的机器学习方法（常导致模型欠拟合或过拟合问题）。本研究提出一种创新方法，利用零样本和小样本生成式预训练Transformer（GPT）分类器实现层次化试题分类，大幅减少对训练数据的依赖，转而通过类人的语言描述来定义类别。借助结构化的Python字典，该方法能够无缝解析考试蓝图的层次结构，实现对试题的多层级逐级分类。基于人工数据的初步仿真验证了该方法的有效性，其F1分数衡量的平均准确率达92.91%。该方法进一步应用于美国家庭医学委员会（ABFM）2022年在职培训考试（ITE）的真实试题，将200道试题按新制定的蓝图在15分钟内完成快速重新分类——传统流程通常需要编辑与医生团队耗时数日。这一创新方法不仅大幅缩短分类时间，还能确保基于原则的一致性分类，减少人为偏差与分歧。通过调整定义即可优化分类的特性，进一步增强了其稳健性与可持续性。