Beam search with masked language models (MLMs) is challenging in part because joint probability distributions over sequences are not readily available, unlike for autoregressive models. However, estimating such distributions has important domain-specific applications such as ancient text restoration and protein engineering. Here we present probabilistically-sound methods for beam search with MLMs. First, we clarify the conditions under which it is theoretically sound to perform text infilling with MLMs using standard beam search. When these conditions fail, we provide a probabilistically-sound modification with no additional computational complexity and demonstrate that it is superior to the aforementioned beam search in the expected conditions. We then present empirical results comparing several infilling approaches with MLMs across several domains.

翻译：使用掩码语言模型（MLMs）进行束搜索具有挑战性，部分原因在于序列的联合概率分布不像自回归模型那样容易获得。然而，估计此类分布对于特定领域应用（如古代文本修复和蛋白质工程）具有重要意义。本文提出了基于掩码语言模型的概率可靠束搜索方法。首先，我们阐明了在何种条件下使用标准束搜索进行MLM文本填充在理论上是可靠的。当这些条件不满足时，我们提出了一种概率可靠的改进方法，该方法不增加计算复杂度，并证明其在预期条件下优于前述束搜索。随后，我们展示了在不同领域中比较多种MLM填充方法的实证结果。