Large language models (LLMs) achieve remarkable performance but demand substantial computational resources, limiting deployment on edge devices and resource-constrained environments. We present TernaryLM, a 132M parameter transformer architecture that employs native 1-bit ternary quantization {-1, 0, +1} during training, achieving significant memory reduction without sacrificing language modeling capability. Unlike post-training quantization approaches that quantize pre-trained full-precision models, TernaryLM learns quantization-aware representations from scratch using straight-through estimators and adaptive per-layer scaling factors. Our experiments demonstrate: (1) validation perplexity of 58.42 on TinyStories; (2) downstream transfer with 82.47 percent F1 on MRPC paraphrase detection; (3) 2.4x memory reduction (498MB vs 1197MB) with comparable inference latency; and (4) stable training dynamics across diverse corpora. We provide layer-wise quantization analysis showing that middle transformer layers exhibit highest compatibility with extreme quantization, informing future non-uniform precision strategies. Our results suggest that native 1-bit training is a promising direction for efficient neural language models. Code is available at https://github.com/1nisharg/TernaryLM-Memory-Efficient-Language-Modeling.

翻译：大型语言模型（LLMs）虽能取得卓越性能，但需要大量计算资源，限制了其在边缘设备和资源受限环境中的部署。本文提出TernaryLM，一种包含1.32亿参数的Transformer架构，在训练过程中采用原生1比特三元量化{-1, 0, +1}，在保持语言建模能力的同时显著降低了内存占用。与对预训练全精度模型进行量化的后训练量化方法不同，TernaryLM通过直通估计器和自适应逐层缩放因子，从零开始学习量化感知的表示。我们的实验表明：（1）在TinyStories数据集上的验证困惑度为58.42；（2）在下游迁移任务中，MRPC复述检测的F1分数达到82.47%；（3）在推理延迟相当的情况下，内存占用减少2.4倍（498MB对比1197MB）；（4）在不同语料库上均表现出稳定的训练动态。我们提供了逐层量化分析，表明中间Transformer层与极端量化具有最高的兼容性，这为未来非均匀精度策略的设计提供了参考。我们的结果表明，原生1比特训练是构建高效神经语言模型的一个有前景的方向。代码可在 https://github.com/1nisharg/TernaryLM-Memory-Efficient-Language-Modeling 获取。