Shannon defined the mutual information between two variables. We illustrate why the true mutual information between a variable and the predictions made by a prediction algorithm is not a suitable measure of prediction quality, but the apparent Shannon mutual information (ASI) is; indeed it is the unique prediction quality measure with either of two very different lists of desirable properties, as previously shown by de Finetti and other authors. However, estimating the uncertainty of the ASI is a difficult problem, because of long and non-symmetric heavy tails to the distribution of the individual values of $j(x,y)=\log\frac{Q_y(x)}{P(x)}$ We propose a Bayesian modelling method for the distribution of $j(x,y)$, from the posterior distribution of which the uncertainty in the ASI can be inferred. This method is based on Dirichlet-based mixtures of skew-Student distributions. We illustrate its use on data from a Bayesian model for prediction of the recurrence time of prostate cancer. We believe that this approach is generally appropriate for most problems, where it is infeasible to derive the explicit distribution of the samples of $j(x,y)$, though the precise modelling parameters may need adjustment to suit particular cases.

翻译：香农定义了两个变量之间的互信息。我们阐述了为何变量与预测算法所做预测之间的真实互信息不适合作为预测质量的度量指标，而表观香农互信息（ASI）则适用；事实上，正如德菲内蒂等学者先前证明的，ASI是具有两种截然不同的理想属性集的唯一预测质量度量。然而，由于个体值$j(x,y)=\log\frac{Q_y(x)}{P(x)}$的分布存在长且非对称的重尾特性，估计ASI的不确定性成为一个难题。我们提出了一种针对$j(x,y)$分布的贝叶斯建模方法，通过其后验分布可以推断ASI的不确定性。该方法基于狄利克雷混合的偏斜学生分布。我们以前列腺癌复发时间的贝叶斯预测模型数据为例演示了该方法的应用。我们认为，尽管具体建模参数可能需要根据特定案例进行调整，但该方法普遍适用于大多数无法推导$j(x,y)$样本显式分布的问题。