Large language models (LLMs) with hundreds of billions of parameters require powerful server-grade GPUs for inference, limiting their practical deployment. To address this challenge, we introduce the outlier-aware weight quantization (OWQ) method, which aims to minimize LLM's footprint through low-precision representation. OWQ prioritizes a small subset of structured weights sensitive to quantization, storing them in high-precision, while applying highly tuned quantization to the remaining dense weights. This sensitivity-aware mixed-precision scheme reduces the quantization error notably, and extensive experiments demonstrate that 3.1-bit models using OWQ perform comparably to 4-bit models optimized by OPTQ. Furthermore, OWQ incorporates a parameter-efficient fine-tuning for task-specific adaptation, called weak column tuning (WCT), enabling accurate task-specific LLM adaptation with minimal memory overhead in the optimized format. OWQ represents a notable advancement in the flexibility, efficiency, and practicality of LLM optimization literature. The source code is available at https://github.com/xvyaward/owq

翻译：拥有数千亿参数的大型语言模型（LLMs）需要强大的服务器级GPU进行推理，这限制了其实际部署。为解决这一挑战，我们提出了一种异常感知的权重量化方法（OWQ），旨在通过低精度表示最小化LLM的计算开销。OWQ优先处理对量化敏感的一小部分结构化权重，将其存储为高精度格式，同时对剩余密集权重应用高度优化的量化策略。这种基于敏感性的混合精度方案显著降低了量化误差，大量实验表明，采用OWQ的3.1位模型性能与经OPTQ优化的4位模型相当。此外，OWQ集成了一种面向任务特定适配的参数高效微调方法——弱列微调（WCT），使得在优化格式下能以最小内存开销实现精确的LLM任务适配。OWQ代表了LLM优化领域在灵活性、效率及实用性方面的显著进展。源代码已开源至https://github.com/xvyaward/owq。