This paper evaluates the ability of Large Language Models (LLMs) to leverage contextual information in the form of structured linguistic representations. Specifically, we examine the impact of encoding both short and long contexts using Abstract Meaning Representation (AMR) structures across a diverse set of language tasks. We perform our analysis using 8-bit quantized and instruction-tuned versions of Llama 3.1 (8B), Phi-3, and Mistral 7B. Our results indicate that, for tasks involving short contexts, augmenting the prompt with the AMR of the original language context often degrades the performance of the underlying LLM. However, for tasks that involve long contexts, such as dialogue summarization in the SAMSum dataset, this enhancement improves LLM performance, for example, by increasing the zero-shot cosine similarity score of Llama 3.1 from 66.2% to 76%. This improvement is more evident in the newer and larger LLMs, but does not extend to the older or smaller ones. In addition, we observe that LLMs can effectively reconstruct the original text from a linearized AMR, achieving a cosine similarity of 81.3% in the best-case scenario.

翻译：本文评估了大型语言模型（LLMs）利用结构化语言表征形式中上下文信息的能力。具体而言，我们通过多样化的语言任务，研究了使用抽象意义表示（AMR）结构编码短上下文与长上下文的影响。我们的分析基于Llama 3.1（8B）、Phi-3和Mistral 7B的8位量化及指令微调版本。实验结果表明，在涉及短上下文的任务中，将原始语言上下文的AMR结构加入提示往往会降低基础LLM的性能。然而，对于涉及长上下文的任务（如SAMSum数据集中的对话摘要），这种增强能提升LLM的性能——例如，将Llama 3.1的零样本余弦相似度得分从66.2%提高至76%。这种改进在更新、规模更大的LLMs中更为显著，但在较旧或较小规模的模型中并未出现。此外，我们观察到LLMs能够有效地从线性化AMR重构原始文本，在最佳情况下可获得81.3%的余弦相似度。