The Large Language Model (LLM) is widely employed for tasks such as intelligent assistants, text summarization, translation, and multi-modality on mobile phones. However, the current methods for on-device LLM deployment maintain slow inference speed, which causes poor user experience. To facilitate high-efficiency LLM deployment on device GPUs, we propose four optimization techniques: (a) a symbolic expression-based approach to support dynamic shape model inference; (b) operator optimizations and execution priority setting to enhance inference speed and reduce phone lagging; (c) an FP4 quantization method termed M0E4 to reduce dequantization overhead; (d) a sub-tensor-based technique to eliminate the need for copying KV cache after LLM inference. Furthermore, we implement these methods in our mobile inference engine, Transformer-Lite, which is compatible with both Qualcomm and MTK processors. We evaluated Transformer-Lite's performance using LLMs with varied architectures and parameters ranging from 2B to 14B. Specifically, we achieved prefill and decoding speeds of 121 token/s and 14 token/s for ChatGLM2 6B, and 330 token/s and 30 token/s for smaller Gemma 2B, respectively. Compared with CPU-based FastLLM and GPU-based MLC-LLM, our engine attains over 10x speedup for the prefill speed and 2~3x speedup for the decoding speed.

翻译：大语言模型（LLM）被广泛应用于手机上的智能助手、文本摘要、翻译及多模态任务。然而，当前端侧LLM部署方法推理速度缓慢，导致用户体验不佳。为实现在设备端GPU上高效部署LLM，我们提出四种优化技术：（a）基于符号表达式的方法，支持动态形状模型推理；（b）算子优化与执行优先级设定，提升推理速度并减少手机卡顿；（c）名为M0E4的FP4量化方法，降低反量化开销；（d）基于子张量的技术，消除LLM推理后复制KV缓存的必要。此外，我们将这些方法实现于移动推理引擎Transformer-Lite中，该引擎兼容高通与联发科处理器。我们使用参数规模从2B至14B、架构各异的LLM评估Transformer-Lite性能。具体而言，针对ChatGLM2 6B模型，我们实现了121 token/s的预填充速度和14 token/s的解码速度；针对较小规模的Gemma 2B模型，则分别达到330 token/s和30 token/s。与基于CPU的FastLLM及基于GPU的MLC-LLM相比，我们的引擎在预填充速度上实现超10倍加速，在解码速度上获得2~3倍加速。