The advent of ultra-low-bit LLM models (1/1.58/2-bit), which match the perplexity and end-task performance of their full-precision counterparts using the same model size, is ushering in a new era of LLM inference for resource-constrained environments such as edge devices and AI PCs. While these quantization advances promise models that are more cost-effective in terms of latency, memory, throughput, and energy consumption, the computational efficiency of state-of-the-art (SOTA) inference runtimes (e.g., bitnet.cpp) used to deploy them remains underexplored. In this work, we take a bottom-up approach: we first design and implement 1-bit and 2-bit microkernels optimized for modern CPUs, achieving peak computational efficiency across a variety of CPU platforms. We integrate these microkernels into a state-of-the-art LLM inference framework, namely PyTorch-TPP, and present end-to-end inference results with 2-bit models that outperform the current SOTA runtime bitnet.cpp by up to 2.2x, and deliver up to 7x speedup compared to the 16-bit model inference. We then extend this work to Intel GPUs where we design and implement mixed precision, 2-bit GEMM kernels, and show their performance to be close to optimal. We integrated our optimized Xe2 kernels in the vLLM framework as a quantization plugin and evaluated end-to-end LLM inference results for a range of LLM models and Xe2 GPUs. Depending on the model and platform, we see a 4x - 8x reduction in GEMM time compared to the BF16 case, and we get up to 6.3x speedup in end-to-end latency compared to the BF16 execution. Our optimized runtime advances the state of LLM inference on AI PCs and Intel Xe GPUs, paving the way for efficient deployment of ultra-low-bit LLM models.

翻译：超低位宽大语言模型（1/1.58/2位）的出现，正在为边缘设备和AI PC等资源受限环境开启大语言模型推理的新纪元。这些模型在保持相同模型规模的前提下，其困惑度与终端任务性能已能媲美全精度模型。尽管此类量化技术的进步有望在延迟、内存占用、吞吐量和能耗方面带来更具成本效益的模型，但用于部署这些模型的最先进推理运行时（例如bitnet.cpp）的计算效率仍未得到充分探索。本研究采用自底向上的方法：首先，我们针对现代CPU设计并实现了经过优化的1位和2位微内核，在多种CPU平台上实现了峰值计算效率。我们将这些微内核集成至最先进的大语言模型推理框架PyTorch-TPP中，并展示了2位模型的端到端推理结果：相较于当前最先进的运行时bitnet.cpp，性能提升最高达2.2倍；与16位模型推理相比，速度提升最高达7倍。随后，我们将此项工作扩展至英特尔GPU，设计并实现了混合精度2位GEMM内核，其性能表现接近理论最优。我们将优化的Xe2内核以量化插件形式集成至vLLM框架中，并对一系列大语言模型及Xe2 GPU进行了端到端推理性能评估。根据模型与平台的不同，相较于BF16基准，我们的方案使GEMM计算时间减少了4至8倍，端到端延迟相较于BF16执行最高提升了6.3倍。我们优化的运行时技术推动了大语言模型在AI PC和英特尔Xe GPU上的推理性能前沿，为超低位宽大语言模型的高效部署铺平了道路。