Large language models (LLMs) have demonstrated remarkable performance and tremendous potential across a wide range of tasks. However, deploying these models has been challenging due to the astronomical amount of model parameters, which requires a demand for large memory capacity and high memory bandwidth. In this paper, we propose an effective approach that can make the deployment of LLMs more efficiently. We support an automatic INT4 weight-only quantization flow and design a special LLM runtime with highly-optimized kernels to accelerate the LLM inference on CPUs. We demonstrate the general applicability of our approach on popular LLMs including Llama2, Llama, GPT-NeoX, and showcase the extreme inference efficiency on CPUs. The code is publicly available at: https://github.com/intel/intel-extension-for-transformers.

翻译：大型语言模型（LLMs）在一系列任务中展现了卓越的性能和巨大潜力。然而，由于模型参数规模极其庞大，需要大容量内存和高内存带宽，部署这些模型一直面临挑战。本文提出一种能更高效部署LLMs的有效方法。我们支持自动INT4权重量化流程，并设计了一套包含高度优化内核的特殊LLM运行时，以加速CPU上的LLM推理。我们在包括Llama2、Llama、GPT-NeoX在内的主流LLMs上验证了该方法的广泛适用性，并展示了其在CPU上的极致推理效率。代码已在以下地址公开：https://github.com/intel/intel-extension-for-transformers。