Large language models (LLMs) have been applied in various applications due to their astonishing capabilities. With advancements in technologies such as chain-of-thought (CoT) prompting and in-context learning (ICL), the prompts fed to LLMs are becoming increasingly lengthy, even exceeding tens of thousands of tokens. To accelerate model inference and reduce cost, this paper presents LLMLingua, a coarse-to-fine prompt compression method that involves a budget controller to maintain semantic integrity under high compression ratios, a token-level iterative compression algorithm to better model the interdependence between compressed contents, and an instruction tuning based method for distribution alignment between language models. We conduct experiments and analysis over four datasets from different scenarios, i.e., GSM8K, BBH, ShareGPT, and Arxiv-March23; showing that the proposed approach yields state-of-the-art performance and allows for up to 20x compression with little performance loss. Our code is available at https://aka.ms/LLMLingua.

翻译：大型语言模型（LLM）因其卓越的能力已被广泛应用于各类场景。随着思维链（CoT）提示和上下文学习（ICL）等技术的进步，输入至LLM的提示文本正变得日益冗长，甚至超过数万词元。为加速模型推理并降低成本，本文提出LLMLingua——一种由粗到细的提示压缩方法，该方法包含：预算控制器以在高压缩比下保持语义完整性、词元级迭代压缩算法以更好建模压缩内容间的相互依赖性，以及基于指令微调的分布对齐方法以协调语言模型间的分布差异。我们在四个不同场景的数据集（即GSM8K、BBH、ShareGPT和Arxiv-March23）上进行实验与分析，结果表明所提方法取得了最先进性能，且能在性能损失极小的情况下实现高达20倍的压缩。我们的代码已开源至https://aka.ms/LLMLingua。