Edge devices such as smartwatches and smart glasses cannot continuously run even the smallest 100M-1B parameter language models due to power and compute constraints, yet cloud inference introduces multi-second latencies that break the illusion of a responsive assistant. We introduce micro language models ($μ$LMs): ultra-compact models (8M-30M parameters) that instantly generate the first 4-8 words of a contextually grounded response on-device, while a cloud model completes it; thus, masking the cloud latency. We show that useful language generation survives at this extreme scale with our models matching several 70M-256M-class existing models. We design a collaborative generation framework that reframes the cloud model as a continuator rather than a respondent, achieving seamless mid-sentence handoffs and structured graceful recovery via three error correction methods when the local opener goes wrong. Empirical results show that $μ$LMs can initiate responses that larger models complete seamlessly, demonstrating that orders-of-magnitude asymmetric collaboration is achievable and unlocking responsive AI for extremely resource-constrained devices. The model checkpoint and demo are available at https://github.com/Sensente/micro_language_model_swen_project.

翻译：边缘设备（如智能手表和智能眼镜）因功耗与计算限制，无法持续运行即便是最小的100M-1B参数语言模型，而云端推理会产生数秒延迟，破坏响应式助手的沉浸体验。我们提出微型语言模型（μLMs）：一种超紧凑模型（8M-30M参数），可在设备端即时生成上下文相关的初始4-8个词，同时由云端模型完成后续内容，从而掩盖云端延迟。实验表明，在此极端规模下，语言生成仍具备实用性——我们的模型性能可媲美多款现有70M-256M类模型。我们设计了一种协同生成框架，将云端模型重新定位为续写器而非应答器，通过三种纠错机制实现句中无缝切换和结构化优雅恢复，以应对本地起始词生成错误。实证结果表明，μLMs能发起可由更大模型无缝完成的响应，证明数量级非对称协同的可行性，为资源极度受限设备解锁响应式AI能力。模型检查点与演示代码已发布于https://github.com/Sensente/micro_language_model_swen_project。