For consumer usage of locally deployed LLMs, the GGUF format and k_quantization are invaluable tools for maintaining the performance of the original model while reducing it to sizes deployable with consumer-grade hardware. The number of bits dedicated to each weight from the original model is reduced based on how important they are thought to be during model inference. This importance is arrived at through the application of an 'importance matrix'-a relatively small text document meant to be representative of the LLM's standard use-cases. In the vast majority of quants available online, this document is primarily written in English. It was therefore an open question whether performance on English language tasks was preserved through the sacrifice of multilingual performance and whether it can be preserved with alternate importance matrices. This article investigates these hypotheses by quantizing Llama3.3 70B on importance matrices written in three languages (English, Norwegian, and Malayalam) and evaluating them on the MixEval dataset in both English and Norwegian. All experiments related to k_quantization yielded non-significant results (In all cases p > 0.237) indicating that current quantization practices do not disproportionately harm multilingual performance.

翻译：对于本地部署的LLM消费级应用而言，GGUF格式与k_量化是宝贵的工具，它们能在将模型压缩至消费级硬件可部署大小的同时，保持原始模型的性能。量化过程根据模型推理中权重的重要性，减少原始模型中每个权重所占用的比特数。这种重要性是通过应用“重要性矩阵”来确定的——这是一个相对较小的文本文件，旨在代表LLM的标准使用场景。目前网络上绝大多数可用的量化版本中，该文档主要使用英文撰写。因此，一个悬而未决的问题是：量化过程是否以牺牲多语言性能为代价来保持英语任务性能，以及是否可以通过替代的重要性矩阵来保持多语言性能。本文通过使用三种语言（英语、挪威语和马拉雅拉姆语）撰写的重要性矩阵对Llama3.3 70B进行量化，并在MixEval数据集上对英语和挪威语任务进行评估，以检验这些假设。所有与k_量化相关的实验均未产生显著结果（所有情况下p值均大于0.237），这表明当前的量化实践并未对多语言性能造成不成比例的损害。