The breakthrough performance of large language models (LLMs) comes with large computational footprints and high deployment costs. In this paper, we progress towards resolving this problem by proposing a new structured compression approach for LLMs, called ZipLM, which provides state-of-the-art compression-vs-accuracy results, while guaranteeing to match a set of (achievable) target speedups on any given target hardware. Specifically, given a task, a model, an inference environment, as well as a set of speedup targets, ZipLM identifies and removes redundancies in the model through iterative structured shrinking of the model's weight matrices. Importantly, ZipLM works in both, the post-training/one-shot and the gradual compression setting, where it produces a set of accurate models in a single run, making it highly-efficient in practice. Our approach is based on new structured pruning and knowledge distillation techniques, and consistently outperforms prior structured compression methods in terms of accuracy-versus-speedup in experiments on BERT- and GPT-family models. In particular, when compressing GPT2 model, it outperforms DistilGPT2 while being 60% smaller and 30% faster. Further, ZipLM matches performance of heavily optimized MobileBERT model, obtained via extensive architecture search, by simply pruning the baseline BERT-large architecture, and outperforms all prior BERT-base compression techniques like CoFi, MiniLM and TinyBERT.

翻译：大型语言模型的突破性性能伴随着巨大的计算开销和高昂的部署成本。本文通过提出一种名为ZipLM的新型结构化压缩方法，朝着解决这一问题迈出关键一步。该方法在任意目标硬件上保证匹配一组（可实现的）加速目标的同时，实现了最先进的压缩-精度权衡。具体而言，给定任务、模型、推理环境以及一组加速目标，ZipLM通过迭代式结构化缩减模型权重矩阵来识别并消除模型冗余。值得关注的是，ZipLM既适用于训练后/单次压缩场景，也适用于渐进式压缩场景——单次运行即可生成一组高精度模型，显著提升实际部署效率。该方法基于新型结构化剪枝与知识蒸馏技术，在BERT族和GPT族模型实验中，始终在精度-加速比权衡上超越现有结构化压缩方法。例如在压缩GPT2模型时，ZipLM较DistilGPT2体积减小60%，速度提升30%，同时保持更优性能。此外，仅通过剪枝基础BERT-large架构，ZipLM即可匹配经大规模架构搜索优化的MobileBERT模型性能，并全面超越CoFi、MiniLM和TinyBERT等现有BERT-base压缩技术。