Transformers have revolutionized the machine learning landscape, gradually making their way into everyday tasks and equipping our computers with ``sparks of intelligence''. However, their runtime requirements have prevented them from being broadly deployed on mobile. As personal devices become increasingly powerful and prompt privacy becomes an ever more pressing issue, we explore the current state of mobile execution of Large Language Models (LLMs). To achieve this, we have created our own automation infrastructure, MELT, which supports the headless execution and benchmarking of LLMs on device, supporting different models, devices and frameworks, including Android, iOS and Nvidia Jetson devices. We evaluate popular instruction fine-tuned LLMs and leverage different frameworks to measure their end-to-end and granular performance, tracing their memory and energy requirements along the way. Our analysis is the first systematic study of on-device LLM execution, quantifying performance, energy efficiency and accuracy across various state-of-the-art models and showcases the state of on-device intelligence in the era of hyperscale models. Results highlight the performance heterogeneity across targets and corroborates that LLM inference is largely memory-bound. Quantization drastically reduces memory requirements and renders execution viable, but at a non-negligible accuracy cost. Drawing from its energy footprint and thermal behavior, the continuous execution of LLMs remains elusive, as both factors negatively affect user experience. Last, our experience shows that the ecosystem is still in its infancy, and algorithmic as well as hardware breakthroughs can significantly shift the execution cost. We expect NPU acceleration, and framework-hardware co-design to be the biggest bet towards efficient standalone execution, with the alternative of offloading tailored towards edge deployments.

翻译：Transformer已彻底改变了机器学习的格局，逐渐进入日常任务领域，为计算机注入了“智能火花”。然而，其运行时需求阻碍了在移动端的广泛部署。随着个人设备性能日益增强，且用户隐私问题日益紧迫，我们探索了大语言模型（LLMs）在移动端执行的当前状态。为此，我们构建了自有自动化基础设施MELT，支持在设备上无头运行和基准测试LLMs，兼容不同模型、设备及框架，包括Android、iOS和Nvidia Jetson设备。我们评估了主流指令微调LLMs，并利用多种框架测量其端到端及细粒度性能，同时追踪其内存与能耗需求。本研究首次系统性分析设备端LLM执行，量化了多种先进模型的性能、能效与准确性，揭示了超大规模模型时代设备端智能的现状。结果表明，不同目标平台存在显著的性能异质性，并证实LLM推理主要受内存限制。量化技术大幅降低了内存需求，使执行变得可行，但带来了不可忽视的准确性损失。从能耗足迹与热行为来看，LLM的持续执行仍难以实现，这两方面因素均对用户体验产生负面影响。最后，我们的经验表明，该生态系统尚处早期阶段，算法与硬件的突破可能显著改变执行成本。我们预计，NPU加速以及框架-硬件协同设计将是实现高效独立执行的最大突破口，而卸载方案则更适用于边缘部署场景。