Assessment is a crucial part of education. Traditional marking is a source of inconsistencies and unconscious bias, placing a high cognitive load on the assessors. An approach to address these issues is comparative judgement (CJ). In CJ, the assessor is presented with a pair of items and is asked to select the better one. Following a series of comparisons, a rank is derived using a ranking model, for example, the BTM, based on the results. While CJ is considered a reliable method for marking, there are concerns around transparency, and the ideal number of pairwise comparisons to generate a reliable estimation of the rank order is not known. Additionally, there have been attempts to generate a method of selecting pairs that should be compared next in an informative manner, but some existing methods are known to have created their own bias within results inflating the reliability metric used. As a result, a random selection approach is usually deployed. We propose a novel Bayesian approach to CJ (BCJ) for determining the ranks of compared items alongside a new way to select the pairs to present to the marker(s) using active learning (AL), addressing the key shortcomings of traditional CJ. Furthermore, we demonstrate how the entire approach may provide transparency by providing the user insights into how it is making its decisions and, at the same time, being more efficient. Results from our experiments confirm that the proposed BCJ combined with entropy-driven AL pair-selection method is superior to other alternatives. We also find that the more comparisons done, the more accurate BCJ becomes, which solves the issue the current method has of the model deteriorating if too many comparisons are performed. As our approach can generate the complete predicted rank distribution for an item, we also show how this can be utilised in devising a predicted grade, guided by the assessor.

翻译：评估是教育的关键环节。传统评分方式存在不一致性和无意识偏见，给评分者带来较高的认知负荷。解决这些问题的方法之一是采用比较判断法（CJ）。在比较判断法中，评估者需面对一组项目并选择其中较优的一个。通过一系列比较，基于评分结果（例如使用BTM模型）推导出排名。尽管比较判断法被认为是一种可靠的评分方法，但其透明度仍存疑虑，且目前尚未明确为获得可靠排名估计所需的最优成对比较次数。此外，已有研究试图生成一种能提供信息量的配对选择方法以确定下一组需比较的项目，但部分现有方法被指在结果中制造了固有偏差，导致信度指标虚高。因此，通常采用随机配对策略。本文提出一种新颖的贝叶斯比较判断方法（BCJ），用于确定被比较项目的排名，同时结合主动学习（AL）技术实现向评分者最优配对推荐，从而解决传统比较判断法的关键缺陷。进一步地，我们展示了该方法如何通过向用户揭示决策过程以提高透明度，同时提升效率。实验结果表明，所提出的BCJ方法结合基于信息熵驱动的主动学习配对策略显著优于其他替代方案。我们还发现，随着比较次数的增加，BCJ的准确性随之提升，这解决了现有方法因过多比较而导致的模型退化问题。由于我们的方法可生成项目的完整预测排名分布，我们进一步展示了如何利用该分布在评估者指导下推导预测评分等级。