Shannon defined the mutual information between two variables. We illustrate why the true mutual information between a variable and the predictions made by a prediction algorithm is not a suitable measure of prediction quality, but the apparent Shannon mutual information (ASI) is; indeed it is the unique prediction quality measure with either of two very different lists of desirable properties, as previously shown by de Finetti and other authors. However, estimating the uncertainty of the ASI is a difficult problem, because of long and non-symmetric heavy tails to the distribution of the individual values of $j(x,y)=\log\frac{Q_y(x)}{P(x)}$ We propose a Bayesian modelling method for the distribution of $j(x,y)$, from the posterior distribution of which the uncertainty in the ASI can be inferred. This method is based on Dirichlet-based mixtures of skew-Student distributions. We illustrate its use on data from a Bayesian model for prediction of the recurrence time of prostate cancer. We believe that this approach is generally appropriate for most problems, where it is infeasible to derive the explicit distribution of the samples of $j(x,y)$, though the precise modelling parameters may need adjustment to suit particular cases.

翻译：香农定义了两个变量之间的互信息。我们论证了变量及其预测算法预测结果之间的真实互信息并不适合作为预测质量的度量，而表观香农互信息（ASI）才是合适的度量；事实上，正如德菲内蒂及其他作者先前所证明的，ASI 是唯一满足两组截然不同的理想性质列表的预测质量度量。然而，由于 $j(x,y)=\log\frac{Q_y(x)}{P(x)}$ 各值的分布存在长尾和非对称厚尾特征，ASI 不确定性的估计是一项难题。我们提出了一种针对 $j(x,y)$ 分布的贝叶斯建模方法，通过其后验分布可推断 ASI 的不确定性。该方法基于狄利克雷混合偏 t 分布。我们以前列腺癌复发时间预测的贝叶斯模型数据为例说明了其应用。我们相信，在无法直接推导 $j(x,y)$ 样本显式分布的大多数问题中，该方法具有普适性，尽管具体建模参数可能需要针对特定情况进行调整。