Matryoshka Quantization (MatQuant) is a recent quantization approach showing that a single integer-quantized model can be served across multiple precisions, by slicing the most significant bits (MSB) at inference time. This enables a single checkpoint to cover a wide range of memory and latency budgets, but renders quantization much more challenging. In particular, the initial MatQuant relies on expensive quantization-aware training (QAT) variants, rather than fast one-shot post training quantization (PTQ), and lacks open-source and kernel support. We address all of these limitations by introducing Post-Training Matryoshka Quantization (MatGPTQ), a new PTQ pipeline that produces a single parent model jointly optimized for multiple target precisions in one-shot, based on a small calibration set. MatGPTQ casts Matryoshka quantization as a multi-precision objective with bit-slicing and cross-bit error compensation, resulting in an algorithm that produces a multi-bit-width, "sliceable" model in a single pass. We also incorporate a new budget-aware search for heterogeneous per-layer bit-witdhs and provide efficient kernels that implement slicing and mixed-precision execution. Across standard LLMs and benchmarks, MatGPTQ preserves high-bit accuracy while substantially improving performance at low-bit-witdh settings. Overall, we establish a new state of the art for Matryoshka-style post-training quantization and make single-checkpoint, multi-precision deployment open and practical. Code is available at https://github.com/IST-DASLab/MatGPTQ.

翻译：套娃量化（MatQuant）是近期提出的一种量化方法，其表明单个整数量化模型可通过在推理时切片最高有效位（MSB）来支持多种精度。这使得单个检查点能够覆盖广泛的内存和延迟预算，但也使量化变得更具挑战性。具体而言，最初的套娃量化依赖于昂贵的量化感知训练（QAT）变体，而非快速的单次训练后量化（PTQ），且缺乏开源和内核支持。我们通过引入训练后套娃量化（MatGPTQ）解决了所有这些限制，这是一种基于小型校准集、通过单次过程生成针对多个目标精度联合优化的单一父模型的新型PTQ流程。MatGPTQ将套娃量化为具有位切片和跨位误差补偿的多精度目标，形成了一种在单次处理中生成多比特宽度、“可切片”模型的算法。我们还引入了针对异构逐层比特宽度的新型预算感知搜索，并提供了实现切片和混合精度执行的高效内核。在标准大语言模型和基准测试中，MatGPTQ在保持高比特精度的同时，显著提升了低比特宽度设置下的性能。总体而言，我们为套娃式训练后量化建立了新的技术标杆，并使单检查点多精度部署变得开放且实用。代码发布于 https://github.com/IST-DASLab/MatGPTQ。