The advent of modern data collection and processing techniques has seen the size, scale, and complexity of data grow exponentially. A seminal step in leveraging these rich datasets for downstream inference is understanding the characteristics of the data which are repeatable -- the aspects of the data that are able to be identified under a duplicated analysis. Conflictingly, the utility of traditional repeatability measures, such as the intraclass correlation coefficient, under these settings is limited. In recent work, novel data repeatability measures have been introduced in the context where a set of subjects are measured twice or more, including: fingerprinting, rank sums, and generalizations of the intraclass correlation coefficient. However, the relationships between, and the best practices among these measures remains largely unknown. In this manuscript, we formalize a novel repeatability measure, discriminability. We show that it is deterministically linked with the correlation coefficient under univariate random effect models, and has desired property of optimal accuracy for inferential tasks using multivariate measurements. Additionally, we overview and systematically compare repeatability statistics using both theoretical results and simulations. We show that the rank sum statistic is deterministically linked to a consistent estimator of discriminability. The power of permutation tests derived from these measures are compared numerically under Gaussian and non-Gaussian settings, with and without simulated batch effects. Motivated by both theoretical and empirical results, we provide methodological recommendations for each benchmark setting to serve as a resource for future analyses. We believe these recommendations will play an important role towards improving repeatability in fields such as functional magnetic resonance imaging, genomics, pharmacology, and more.

翻译：现代数据收集与处理技术的兴起，使得数据的规模、范围和复杂性呈指数级增长。利用这些丰富数据集进行下游推断的一个关键步骤，是理解数据中可重复的特征——即在重复分析中能够被识别的数据特性。然而，传统可重复性度量（如组内相关系数）在此类场景下的效用有限。近期研究在针对同一组对象进行两次或多次测量的背景下，引入了多种新型数据可重复性度量，包括：指纹识别法、秩和统计量以及组内相关系数的推广形式。然而，这些度量之间的关系及其最佳实践方案在很大程度上仍不明确。本文提出了一种新的可重复性度量——可区分性，并对其进行了形式化定义。我们证明在单变量随机效应模型下，该度量与相关系数存在确定性关联，且在使用多变量测量的推断任务中具备最优精度的理想特性。此外，我们通过理论结果与模拟实验，系统性地综述和比较了各类可重复性统计量。研究表明，秩和统计量与可区分性的一致估计量存在确定性关联。我们通过数值模拟，在高斯与非高斯设定下（包含与不包含模拟批次效应的情况），比较了基于这些度量构建的置换检验的统计功效。基于理论与实证结果，我们为各基准设定提供了方法学建议，以作为未来分析的研究资源。我们相信这些建议将对提升功能磁共振成像、基因组学、药理学等领域的可重复性发挥重要作用。