Do LMs infer the semantics of text from co-occurrence patterns in their training data? Merrill et al. (2022) argue that, in theory, probabilities predicted by an optimal LM encode semantic information about entailment relations, but it is unclear whether neural LMs trained on corpora learn entailment in this way because of strong idealizing assumptions made by Merrill et al. In this work, we investigate whether their theory can be used to decode entailment judgments from neural LMs. We find that a test similar to theirs can decode entailment relations between natural sentences, well above random chance, though not perfectly, across many datasets and LMs. This suggests LMs implicitly model aspects of semantics to predict semantic effects on sentence co-occurrence patterns. However, we find the test that predicts entailment in practice works in the opposite direction to the theoretical test. We thus revisit the assumptions underlying the original test, finding its derivation did not adequately account for redundancy in human-written text. We argue that correctly accounting for redundancy related to explanations might derive the observed flipped test and, more generally, improve linguistic theories of human speakers.

翻译：语言模型能否从训练数据中的共现模式推断文本语义？Merrill等人（2022）在理论上论证，最优语言模型预测的概率可编码蕴含关系的语义信息，但由于该研究采用了较强的理想化假设，尚不清楚经语料训练的神经语言模型是否通过此路径习得蕴含关系。本文探究该理论是否可用于从神经语言模型中解码蕴含判断。实验表明，与Merrill等人相似的测试方法虽无法达到完美预测，但在多个数据集和语言模型上，均能显著高于随机水平地解码自然语句间的蕴含关系。这暗示语言模型通过隐式建模语义来预测句子共现模式中的语义效应。然而，我们发现实际预测蕴含的测试方向与理论测试恰好相反。由此重新审视原始测试的假设基础，发现其推导过程未充分考虑人类撰写文本中的冗余现象。我们论证指出，合理纳入与解释相关的冗余因素，不仅可推导出观察到的反向测试现象，更可普遍改善关于人类说话者的语言学理论。