Estimating the log-likelihood of a given sentence under an autoregressive language model is straightforward: one can simply apply the chain rule and sum the log-likelihood values for each successive token. However, for masked language models (MLMs), there is no direct way to estimate the log-likelihood of a sentence. To address this issue, Salazar et al. (2020) propose to estimate sentence pseudo-log-likelihood (PLL) scores, computed by successively masking each sentence token, retrieving its score using the rest of the sentence as context, and summing the resulting values. Here, we demonstrate that the original PLL method yields inflated scores for out-of-vocabulary words and propose an adapted metric, in which we mask not only the target token, but also all within-word tokens to the right of the target. We show that our adapted metric (PLL-word-l2r) outperforms both the original PLL metric and a PLL metric in which all within-word tokens are masked. In particular, it better satisfies theoretical desiderata and better correlates with scores from autoregressive models. Finally, we show that the choice of metric affects even tightly controlled, minimal pair evaluation benchmarks (such as BLiMP), underscoring the importance of selecting an appropriate scoring metric for evaluating MLM properties.

翻译：估计自回归语言模型下给定句子的对数似然是直接的：只需应用链式法则，对每个连续词元的对数似然值求和。然而，对于掩码语言模型（MLM），并没有直接的方法来估计句子的对数似然。为解决此问题，Salazar等人（2020）提出估计句子伪对数似然（PLL）分数，通过依次掩码每个句子词元，利用句子其余部分作为上下文检索其分数，并对结果值求和。在此，我们证明原始PLL方法会导致词汇外单词的分数虚高，并提出一种改进指标，其中不仅掩码目标词元，还掩码目标右侧的所有词内词元。我们表明，改进指标（PLL-word-l2r）优于原始PLL指标以及掩码所有词内词元的PLL指标。特别地，它能更好地满足理论要求，并与自回归模型的分数有更强的相关性。最后，我们证明指标的选择甚至会影响严格控制的极小对评估基准（如BLiMP），这突显了选择适当评分指标以评估MLM属性的重要性。