Transformers have revolutionized the machine learning landscape, gradually making their way into everyday tasks and equipping our computers with ``sparks of intelligence''. However, their runtime requirements have prevented them from being broadly deployed on mobile. As personal devices become increasingly powerful and prompt privacy becomes an ever more pressing issue, we explore the current state of mobile execution of Large Language Models (LLMs). To achieve this, we have created our own automation infrastructure, MELT, which supports the headless execution and benchmarking of LLMs on device, supporting different models, devices and frameworks, including Android, iOS and Nvidia Jetson devices. We evaluate popular instruction fine-tuned LLMs and leverage different frameworks to measure their end-to-end and granular performance, tracing their memory and energy requirements along the way. Our analysis is the first systematic study of on-device LLM execution, quantifying performance, energy efficiency and accuracy across various state-of-the-art models and showcases the state of on-device intelligence in the era of hyperscale models. Results highlight the performance heterogeneity across targets and corroborates that LLM inference is largely memory-bound. Quantization drastically reduces memory requirements and renders execution viable, but at a non-negligible accuracy cost. Drawing from its energy footprint and thermal behavior, the continuous execution of LLMs remains elusive, as both factors negatively affect user experience. Last, our experience shows that the ecosystem is still in its infancy, and algorithmic as well as hardware breakthroughs can significantly shift the execution cost. We expect NPU acceleration, and framework-hardware co-design to be the biggest bet towards efficient standalone execution, with the alternative of offloading tailored towards edge deployments.

翻译：Transformer已彻底改变了机器学习领域，逐渐融入日常任务，为计算机赋予“智能火花”。然而，其运行时需求使其难以在移动设备上广泛部署。随着个人设备性能日益强大，且隐私问题愈发紧迫，我们探索了大语言模型（LLM）在移动端执行的现状。为此，我们构建了自动化基础设施MELT，支持设备上LLM的无头执行与基准测试，涵盖不同模型、设备及框架，包括Android、iOS和Nvidia Jetson设备。我们评估了流行的指令微调LLM，利用不同框架测量其端到端及细粒度性能，并追踪内存与能耗需求。本研究是首个针对设备上LLM执行的系统性分析，量化了多种先进模型的性能、能效与准确性，揭示了超大规模模型时代设备端智能的现状。结果显示目标间存在显著性能异质性，并证实LLM推理主要受内存限制。量化大幅降低了内存需求，使执行可行，但带来了不可忽视的准确性代价。从能耗与热行为来看，LLM的持续执行仍难以实现，这两方面均对用户体验产生负面影响。最后，我们的经验表明，该生态系统尚处初期阶段，算法与硬件突破可能显著改变执行成本。我们预计NPU加速及框架-硬件协同设计将成为高效独立执行的最大突破点，而卸载方案则更适用于边缘部署场景。