We propose to extract meaning representations from autoregressive language models by considering the distribution of all possible trajectories extending an input text. This strategy is prompt-free, does not require fine-tuning, and is applicable to any pre-trained autoregressive model. Moreover, unlike vector-based representations, distribution-based representations can also model asymmetric relations (e.g., direction of logical entailment, hypernym/hyponym relations) by using algebraic operations between likelihood functions. These ideas are grounded in distributional perspectives on semantics and are connected to standard constructions in automata theory, but to our knowledge they have not been applied to modern language models. We empirically show that the representations obtained from large models align well with human annotations, outperform other zero-shot and prompt-free methods on semantic similarity tasks, and can be used to solve more complex entailment and containment tasks that standard embeddings cannot handle. Finally, we extend our method to represent data from different modalities (e.g., image and text) using multimodal autoregressive models. Our code is available at: https://github.com/tianyu139/meaning-as-trajectories

翻译：我们提出通过考虑扩展输入文本的所有可能轨迹分布，从自回归语言模型中提取意义表示。该策略无需提示、无需微调，且适用于任何预训练的自回归模型。此外，与基于向量的表示不同，基于分布的表示可以通过似然函数之间的代数运算来建模非对称关系（例如，逻辑蕴含的方向、上下位关系）。这些思想基于语义的分布视角，并与自动机理论中的标准构造相关联，但据我们所知，尚未应用于现代语言模型。我们通过实验证明，从大模型中获得的表示与人类标注高度一致，在语义相似性任务上优于其他零样本和无提示方法，并且可用于解决标准嵌入无法处理的更复杂的蕴含和包容任务。最后，我们将方法扩展到使用多模态自回归模型表示不同模态的数据（如图像和文本）。我们的代码开源在：https://github.com/tianyu139/meaning-as-trajectories