Which neural networks are similar is a fundamental question for both machine learning and neuroscience. Our novel method compares representations based on Bayesian statistics about linear readouts from the representations. Concretely, we suggest to use the total variation distance or Jensen-Shannon distance between prior predictive distributions to compare representations. The prior predictive distribution is a full description of the inductive bias and generalization of a model in Bayesian statistics, making it a great basis for comparisons. As Jensen-Shannon distance and total variation distance are metrics our dissimilarity measures are pseudo-metrics for representations. For a linear readout, our metrics just depend on the linear kernel matrix of the representations. Thus, our metrics connects linear read-out based comparisons to kernel based metrics like centered kernel alignment and representational similarity analysis. We apply our new metrics to deep neural networks trained on ImageNet-1k. Our new metrics can be computed efficiently including a stochastic gradient without dimensionality reductions of the representations. It broadly agrees with existing metrics, but is more stringent. It varies less across different random image samples, and it measures how well two representations could be distinguished based on a linear read out. Thus our metric nicely extends our toolkit for comparing representations.

翻译：神经网络间的相似性问题是机器学习和神经科学领域的一个基础性问题。我们提出了一种基于贝叶斯统计的新型表征比较方法，该方法通过分析从表征中提取的线性读出信息进行比较。具体而言，我们建议使用先验预测分布之间的总变差距离或詹森-香农距离来比较表征。在贝叶斯统计中，先验预测分布完整描述了模型的归纳偏好和泛化特性，因此成为比较表征的理想基础。由于詹森-香农距离和总变差距离均为度量标准，我们的差异性度量构成了表征的伪度量。对于线性读出器，我们的度量仅取决于表征的线性核矩阵。因此，我们的度量将基于线性读出的比较方法与核度量方法（如中心核对齐和表征相似性分析）联系起来。我们将新度量应用于在ImageNet-1k数据集上训练的深度神经网络。新度量可高效计算（包括随机梯度计算），且无需对表征进行降维处理。该方法与现有度量标准总体一致，但更为严格：其在不同随机图像样本间的波动更小，并能有效度量基于线性读出器区分两种表征的能力。因此，我们的度量方法显著扩展了表征比较的工具集。