The growing adoption of Apple Silicon for machine learning development has created demand for efficient inference solutions that leverage its unique unified memory architecture. However, existing tools either lack native optimization (PyTorch MPS) or focus solely on text models (llama.cpp), leaving multimodal workloads underserved. We present vllm-mlx, a framework for efficient LLM and MLLM inference on Apple Silicon built natively on MLX. For text models, we achieve 21% to 87% higher throughput than llama.cpp across models ranging from Qwen3-0.6B to Nemotron-30B, while providing continuous batching that scales to 4.3x aggregate throughput at 16 concurrent requests. For multimodal models, we introduce content-based prefix caching that eliminates redundant vision encoding by identifying identical images through content hashing, regardless of input format. Our evaluation on Apple M4 Max demonstrates throughput of up to 525 tokens per second on text models and 28x speedup on repeated image queries, reducing multimodal latency from 21.7 seconds to under 1 second. Video analysis with up to 64 frames achieves 24.7x cache speedup. We release our implementation as open source to support efficient inference on consumer Apple hardware.

翻译：苹果芯片在机器学习开发中的日益普及，催生了对利用其独特统一内存架构的高效推理解决方案的需求。然而，现有工具要么缺乏原生优化（如 PyTorch MPS），要么仅专注于文本模型（如 llama.cpp），导致多模态工作负载未能得到充分支持。本文提出了 vllm-mlx，这是一个在 MLX 上原生构建的、用于在苹果芯片上高效进行 LLM 和 MLLM 推理的框架。对于文本模型，在从 Qwen3-0.6B 到 Nemotron-30B 的不同模型上，我们实现了比 llama.cpp 高出 21% 至 87% 的吞吐量，同时提供了连续批处理功能，在 16 个并发请求下可扩展至 4.3 倍的聚合吞吐量。对于多模态模型，我们引入了基于内容的前缀缓存技术，该技术通过内容哈希识别相同图像（无论输入格式如何），从而消除了冗余的视觉编码。我们在苹果 M4 Max 上的评估表明，文本模型的吞吐量最高可达每秒 525 个令牌，在重复图像查询上实现了 28 倍的加速，将多模态延迟从 21.7 秒降低到 1 秒以下。对多达 64 帧的视频分析实现了 24.7 倍的缓存加速。我们将实现开源发布，以支持在消费级苹果硬件上进行高效推理。