Comparative Judgement (CJ) provides an alternative assessment approach by evaluating work holistically rather than breaking it into discrete criteria. This method leverages human ability to make nuanced comparisons, yielding more reliable and valid assessments. CJ aligns with real-world evaluations, where overall quality emerges from the interplay of various elements. However, rubrics remain widely used in education, offering structured criteria for grading and detailed feedback. This creates a gap between CJ's holistic ranking and the need for criterion-based performance breakdowns. This paper addresses this gap using a Bayesian approach. We build on Bayesian CJ (BCJ) by Gray et al., which directly models preferences instead of using likelihoods over total scores, allowing for expected ranks with uncertainty estimation. Their entropy-based active learning method selects the most informative pairwise comparisons for assessors. We extend BCJ to handle multiple independent learning outcome (LO) components, defined by a rubric, enabling both holistic and component-wise predictive rankings with uncertainty estimates. Additionally, we propose a method to aggregate entropies and identify the most informative comparison for assessors. Experiments on synthetic and real data demonstrate our method's effectiveness. Finally, we address a key limitation of BCJ, which is the inability to quantify assessor agreement. We show how to derive agreement levels, enhancing transparency in assessment.

翻译：比较判断（Comparative Judgement, CJ）通过整体性评估作品而非将其分解为离散准则，提供了一种替代性的评估方法。该方法利用人类进行细致比较的能力，从而产生更可靠和有效的评估结果。CJ与现实世界的评估方式相符，其中整体质量源于多种要素的相互作用。然而，评分准则在教育领域仍被广泛使用，为评分和详细反馈提供了结构化的标准。这在CJ的整体性排名与基于准则的表现分解需求之间造成了差距。本文采用贝叶斯方法来解决这一差距。我们在Gray等人提出的贝叶斯CJ（Bayesian CJ, BCJ）基础上进行构建，该模型直接对偏好进行建模，而非使用总分的似然函数，从而能够获得带有不确定性估计的期望排名。他们基于信息熵的主动学习方法为评估者选择了信息量最大的成对比较。我们将BCJ扩展至处理多个独立的学习成果（Learning Outcome, LO）组件，这些组件由评分准则定义，从而能够同时获得带有不确定性估计的整体性和组件层面的预测性排名。此外，我们提出了一种聚合信息熵并识别对评估者最具信息量的比较的方法。在合成数据和真实数据上的实验证明了我们方法的有效性。最后，我们解决了BCJ的一个关键局限性，即无法量化评估者间的一致性。我们展示了如何推导一致性水平，从而增强了评估的透明度。