In several observational contexts where different raters evaluate a set of items, it is common to assume that all raters draw their scores from the same underlying distribution. However, a plenty of scientific works have evidenced the relevance of individual variability in different type of rating tasks. To address this issue the intra-class correlation coefficient (ICC) has been used as a measure of variability among raters within the Hierarchical Linear Models approach. A common distributional assumption in this setting is to specify hierarchical effects as independent and identically distributed from a normal with the mean parameter fixed to zero and unknown variance. The present work aims to overcome this strong assumption in the inter-rater agreement estimation by placing a Dirichlet Process Mixture over the hierarchical effects' prior distribution. A new nonparametric index $\lambda$ is proposed to quantify raters polarization in presence of group heterogeneity. The model is applied on a set of simulated experiments and real world data. Possible future directions are discussed.

翻译：在多位评分者对一组项目进行评分的观察性研究中，通常假设所有评分者都从相同的潜在分布中获取评分。然而，大量科学工作已证明个体差异在不同类型评分任务中的重要性。为解决此问题，组内相关系数（ICC）被用作层次线性模型方法中评分者间变异性的度量指标。在此框架下，常见的分布假设是将层次效应设定为服从均值为零且方差未知的独立同分布正态分布。本研究旨在通过将狄利克雷过程混合模型（Dirichlet Process Mixture）应用于层次效应的先验分布，克服评分者间一致性估计中的这一强假设。我们提出一个新的非参数指标 $\lambda$，用于量化存在群体异质性时的评分者极化程度。该模型在模拟实验和真实世界数据上进行了应用，并讨论了未来可能的研究方向。