We propose to extract meaning representations from autoregressive language models by considering the distribution of all possible trajectories extending an input text. This strategy is prompt-free, does not require fine-tuning, and is applicable to any pre-trained autoregressive model. Moreover, unlike vector-based representations, distribution-based representations can also model asymmetric relations (e.g., direction of logical entailment, hypernym/hyponym relations) by using algebraic operations between likelihood functions. These ideas are grounded in distributional perspectives on semantics and are connected to standard constructions in automata theory, but to our knowledge they have not been applied to modern language models. We empirically show that the representations obtained from large models align well with human annotations, outperform other zero-shot and prompt-free methods on semantic similarity tasks, and can be used to solve more complex entailment and containment tasks that standard embeddings cannot handle. Finally, we extend our method to represent data from different modalities (e.g., image and text) using multimodal autoregressive models.

翻译：我们提出从自回归语言模型中提取意义表示的方法，通过考虑扩展输入文本的所有可能轨迹的分布来实现。该策略无需提示词、无需微调，适用于任何预训练的自回归模型。此外，与基于向量的表示不同，基于分布的表示能够通过似然函数间的代数运算建模不对称关系（例如逻辑蕴含方向、上位词/下位词关系）。这些思想植根于语义的分布视角，并与自动机理论中的标准构造相关联，但据我们所知尚未被应用于现代语言模型。实验表明，从大型模型获得的表示与人工标注高度吻合，在语义相似度任务上优于其他零样本和无提示方法，并能解决标准嵌入无法处理的更复杂的蕴含和包含任务。最后，我们将方法扩展到使用多模态自回归模型表示不同模态数据（如图像和文本）。