Language models are often trained on text alone, without additional grounding. There is debate as to how much of natural language semantics can be inferred from such a procedure. We prove that entailment judgments between sentences can be extracted from an ideal language model that has perfectly learned its target distribution, assuming the training sentences are generated by Gricean agents, i.e., agents who follow fundamental principles of communication from the linguistic theory of pragmatics. We also show entailment judgments can be decoded from the predictions of a language model trained on such Gricean data. Our results reveal a pathway for understanding the semantic information encoded in unlabeled linguistic data and a potential framework for extracting semantics from language models.

翻译：语言模型通常仅基于文本进行训练，缺乏额外的具身基础。关于此类训练过程能在多大程度上推断自然语言语义，学界存在争议。我们证明，假设训练语句由遵循格里森原则的智能体（即遵循语言语用学理论中基本交际原则的智能体）生成，则可以从完美学习目标分布的理想语言模型中提取句子间的蕴涵判断。我们还证明，可从基于此类格里森数据训练的语言模型预测结果中解码出蕴涵判断。该研究结果揭示了理解无标注语言数据中编码的语义信息的路径，并为从语言模型中提取语义提供了潜在框架。