Large, pretrained language models infer powerful representations that encode rich semantic and syntactic content, albeit implicitly. In this work we introduce a novel neural language model that enforces, via inductive biases, explicit relational structures which allow for compositionality onto the output representations of pretrained language models. Specifically, the model encodes sentences into sequences of symbols (composed representations), which correspond to the nodes visited by biased random walkers on a global latent graph, and infers the posterior distribution of the latter. We first demonstrate that the model is able to uncover ground-truth graphs from artificially generated datasets of random token sequences. Next, we leverage pretrained BERT and GPT-2 language models as encoder and decoder, respectively, to infer networks of symbols (schemata) from natural language datasets. Our experiments show that (i) the inferred symbols can be interpreted as encoding different aspects of language, as e.g. topics or sentiments, and that (ii) GPT-like models can effectively be conditioned on symbolic representations. Finally, we explore training autoregressive, random walk ``reasoning" models on schema networks inferred from commonsense knowledge databases, and using the sampled paths to enhance the performance of pretrained language models on commonsense If-Then reasoning tasks.

翻译：大型预训练语言模型能够推断出编码丰富语义和句法内容的强大表征，尽管这些表征是隐式的。本文提出一种新型神经语言模型，通过归纳偏置强制引入显式关系结构，使得预训练语言模型的输出表征具有组合性。具体而言，该模型将句子编码为符号序列（组合表征），这些符号对应于全局潜在图上偏置随机游走所访问的节点，并推断后者的后验分布。我们首先证明，该模型能从人工生成的随机标记序列数据集中还原真实图结构。随后，利用预训练的BERT和GPT-2语言模型分别作为编码器和解码器，从自然语言数据集中推断符号网络（模式）。实验表明：（i）推断出的符号可被解释为语言不同方面（如主题或情感）的编码；（ii）GPT类模型能有效基于符号表征进行条件生成。最后，我们探索在基于常识知识数据库推断出的模式网络上训练自回归随机游走“推理”模型，并利用采样路径增强预训练语言模型在常识性“如果-那么”推理任务上的性能。