Large language models (LLMs) have been applied in various applications due to their astonishing capabilities. With advancements in technologies such as chain-of-thought (CoT) prompting and in-context learning (ICL), the prompts fed to LLMs are becoming increasingly lengthy, even exceeding tens of thousands of tokens. To accelerate model inference and reduce cost, this paper presents LLMLingua, a coarse-to-fine prompt compression method that involves a budget controller to maintain semantic integrity under high compression ratios, a token-level iterative compression algorithm to better model the interdependence between compressed contents, and an instruction tuning based method for distribution alignment between language models. We conduct experiments and analysis over four datasets from different scenarios, i.e., GSM8K, BBH, ShareGPT, and Arxiv-March23; showing that the proposed approach yields state-of-the-art performance and allows for up to 20x compression with little performance loss. Our code is available at https://aka.ms/LLMLingua.

翻译：大语言模型（LLMs）凭借其卓越能力已在各类应用中广泛部署。随着思维链（CoT）提示和上下文学习（ICL）等技术的进步，输入给LLMs的提示文本正变得愈发冗长，甚至超过数万词元。为加速模型推理并降低成本，本文提出LLMLingua——一种从粗到细的提示压缩方法，该方法包含：在高压缩比下维持语义完整性的预算控制器、能更好模拟压缩内容间依赖关系的词元级迭代压缩算法，以及基于指令微调的语言模型分布对齐技术。我们在四个不同场景数据集（GSM8K、BBH、ShareGPT和Arxiv-March23）上开展实验与分析，结果表明所提方法实现了最先进的性能，在仅产生微小性能损失的前提下支持高达20倍的压缩率。相关代码已开源至https://aka.ms/LLMLingua。