Large Language Models (LLMs) have been emerging as prominent AI models for solving many natural language tasks due to their high performance (e.g., accuracy) and capabilities in generating high-quality responses to the given inputs. However, their large computational cost, huge memory footprints, and high processing power/energy make it challenging for their embedded deployments. Amid several tinyLLMs, recent works have proposed spike-driven language models (SLMs) for significantly reducing the processing power/energy of LLMs. However, their memory footprints still remain too large for low-cost and resource-constrained embedded devices. Manual quantization approach may effectively compress SLM memory footprints, but it requires a huge design time and compute power to find the quantization setting for each network, hence making this approach not-scalable for handling different networks, performance requirements, and memory budgets. To bridge this gap, we propose QSLM, a novel framework that performs automated quantization for compressing pre-trained SLMs, while meeting the performance and memory constraints. To achieve this, QSLM first identifies the hierarchy of the given network architecture and the sensitivity of network layers under quantization, then employs a tiered quantization strategy (e.g., global-, block-, and module-level quantization) while leveraging a multi-objective performance-and-memory trade-off function to select the final quantization setting. Experimental results indicate that our QSLM reduces memory footprint by up to 86.5%, reduces power consumption by up to 20%, maintains high performance across different tasks (i.e., by up to 84.4% accuracy of sentiment classification on the SST-2 dataset and perplexity score of 23.2 for text generation on the WikiText-2 dataset) close to the original non-quantized model while meeting the performance and memory constraints.

翻译：大型语言模型因其高精度及生成高质量回应的能力，已成为解决诸多自然语言任务的主流AI模型。然而，其高昂的计算成本、庞大的内存占用及高功耗给嵌入式部署带来了巨大挑战。在众多轻量级语言模型中，近期研究提出脉冲驱动语言模型以显著降低能耗。但这些模型的内存占用对低成本、资源受限的嵌入式设备而言仍显过大。手动量化方法虽能有效压缩脉冲驱动语言模型的内存占用，却需耗费大量设计时间和计算资源为每个网络寻找量化配置，导致该方法无法扩展至处理不同网络、性能需求及内存预算的场景。为解决这一矛盾，我们提出QSLM——一种在满足性能与内存约束的前提下，对预训练脉冲驱动语言模型进行自动量化压缩的新型框架。该框架首先识别给定网络架构的层级特性及网络层对量化的敏感度，随后采用分层量化策略（如全局级、块级及模块级量化），并基于多目标性能-内存权衡函数选取最终量化配置。实验结果表明，QSLM在满足性能与内存约束的同时，可将内存占用最高压缩86.5%，功耗最高降低20%，并在不同任务中保持接近原始非量化模型的高性能（在SST-2数据集上情感分类准确率最高达84.4%，在WikiText-2数据集上文本生成的困惑度得分为23.2）。