Despite the growing prevalence of large language model (LLM) architectures, a crucial concern persists regarding their energy and power consumption, which still lags far behind the remarkable energy efficiency of the human brain. Recent strides in spiking language models (LM) and transformer architectures aim to address this concern by harnessing the spiking activity of biological neurons to enhance energy/power efficiency. Doubling down on the principles of model quantization and energy efficiency, this paper proposes the development of a novel binary/ternary (1/1.58-bit) spiking LM architecture. Achieving scalability comparable to a deep spiking LM architecture is facilitated by an efficient knowledge distillation technique, wherein knowledge from a non-spiking full-precision "teacher" model is transferred to an extremely weight quantized spiking "student" LM. Our proposed model represents a significant advancement as the first-of-its-kind 1/1.58-bit spiking LM, and its performance is rigorously evaluated on multiple text classification tasks of the GLUE benchmark.

翻译：尽管大型语言模型（LLM）架构日益普及，但其能量和功耗问题仍是一个关键关切，与人类大脑卓越的能效相比仍相去甚远。近年来，脉冲语言模型（LM）和Transformer架构通过利用生物神经元的脉冲活动来提升能效，旨在解决这一问题。基于模型量化和能效优化的双重原则，本文提出了一种新型二值/三值（1/1.58比特）脉冲LM架构。通过高效的知识蒸馏技术（将非脉冲全精度"教师"模型的知识迁移至权重极端量化的脉冲"学生"LM），实现了与深度脉冲LM架构相媲美的可扩展性。所提出的模型作为首个1/1.58比特脉冲LM取得了重大突破，并在GLUE基准的多个文本分类任务上对其性能进行了严格评估。