We formulate three generalized Bayesian models for analyzing interrater and intrarater reliability in the presence of multilevel data. Stan implementations of these models provide new estimates of interrater and intrarater reliability. We also derive formulas for calculating marginal correlations under each of the three models. Comparisons of the kappa estimates and marginal correlations across the different models are presented from two real-world datasets. Simulations demonstrate properties of the different measures of agreement under different model assumptions.

翻译：本文针对多层级数据背景下的评估者间与评估者内信度分析，构建了三种广义贝叶斯模型。这些模型的 Stan 实现为评估者间与评估者内信度提供了新的估计方法。我们还推导了三种模型下计算边际相关性的公式。通过两个真实数据集，比较了不同模型间的 kappa 估计值与边际相关性。仿真实验展示了不同模型假设下各类一致性度量的特性。