Large Multimodal Models (LMMs) are inherently modular, consisting of vision and audio encoders, projectors, and large language models. Yet, they are almost always executed monolithically, which underutilizes the heterogeneous accelerators (NPUs, GPUs, DSPs) in modern SoCs and leads to high end-to-end latency. In this paper, we present NANOMIND, a hardware--software co-design inference framework for Large Multimodal Models (LMMs) that breaks large models into modular ``bricks'' (vision, language, audio, etc.) and maps each to its ideal accelerator. The key insight is that large models can be broken into modular components and scheduled to run on the most appropriate compute units. It performs module-level dynamic offloading across accelerators on unified-memory SoCs. By combining customized hardware design, system-level scheduling, and optimized low-bit computation kernels, we demonstrate our framework with a compact, battery-powered device capable of running LMMs entirely on device. This prototype functions as a self-contained intelligent assistant that requires no network connectivity, while achieving higher throughput and superior power efficiency under strict resource constraints. The design further bypasses CPU bottlenecks and reduces redundant memory usage through token-aware buffer management and module-level coordination. Our system outperforms existing implementations in resource efficiency, cutting energy consumption by 42.3\% and GPU memory usage by 11.2\%. This enables a battery-powered device to run LLaVA-OneVision with a camera for nearly 20.8 hours.

翻译：大型多模态模型（LMMs）本质上是模块化的，由视觉与音频编码器、投影器以及大型语言模型构成。然而，它们几乎总是以整体方式执行，这未能充分利用现代片上系统（SoCs）中的异构加速器（NPUs、GPUs、DSPs），并导致较高的端到端延迟。本文提出NANOMIND，一种面向大型多模态模型（LMMs）的软硬件协同设计推理框架，它将大型模型拆解为模块化的“砖块”（视觉、语言、音频等），并将每个模块映射到其理想的加速器上。其核心洞见在于，大型模型可被分解为模块化组件，并调度至最合适的计算单元上运行。该框架在统一内存架构的SoCs上实现了跨加速器的模块级动态卸载。通过结合定制化硬件设计、系统级调度以及优化的低比特计算内核，我们在一款紧凑型电池供电设备上演示了该框架，该设备能够完全在本地运行LMMs。此原型机作为一个自包含的智能助手，无需网络连接，同时在严格资源限制下实现了更高的吞吐量与更优的能效。该设计进一步通过令牌感知的缓冲区管理与模块级协同，绕过了CPU瓶颈并减少了冗余内存使用。我们的系统在资源效率上超越了现有实现，能耗降低了42.3%，GPU内存使用减少了11.2%。这使得一款电池供电设备能够搭载摄像头运行LLaVA-OneVision模型近20.8小时。