Multimodal Large Language Models (MLLMs) have achieved remarkable progress but incur substantial computational overhead and energy consumption during inference, limiting deployment in resource-constrained environments. Spiking Neural Networks (SNNs), with their sparse event-driven computation, offer inherent energy efficiency advantages on neuromorphic hardware, yet extending them to MLLMs faces two key challenges: heterogeneous modalities make uniform spike encoding insufficient, and high-resolution image inputs amplify timestep unfolding overhead. We propose SpikeMLLM, the first spike-based framework for MLLMs, which unifies existing ANN quantization methods in the spiking representation space and incorporates Modality-Specific Temporal Scales (MSTS) guided by Modality Evolution Discrepancy (MED) and Temporally Compressed LIF (TC-LIF) for timestep compression from T=L-1 to T=log2(L)-1. Experiments on four representative MLLMs across diverse multimodal benchmarks show that SpikeMLLM maintains near-lossless performance under aggressive timestep compression (Tv/Tt=3/4), with average gaps of only 0.72% and 1.19% relative to the FP16 baseline on InternVL2-8B and Qwen2VL-72B. We further develop a dedicated RTL accelerator tailored to the spike-driven datapath, observing 9.06x higher throughput and 25.8x better power efficiency relative to an FP16 GPU baseline under a deployment-oriented co-design setting, suggesting the promise of algorithm-hardware co-design for efficient multimodal intelligence.

翻译：多模态大语言模型（MLLMs）虽取得了显著进展，但在推理过程中会带来巨大的计算开销和能耗，限制了其在资源受限环境中的部署。脉冲神经网络（SNNs）凭借其稀疏的事件驱动计算特性，在神经形态硬件上具有固有的能效优势，然而将其扩展到MLLMs面临两个关键挑战：异质模态使得统一的脉冲编码效果不足，以及高分辨率图像输入会放大时间步展开的开销。我们提出SpikeMLLM——首个基于脉冲的MLLM框架，它统一了脉冲表示空间中的现有ANN量化方法，并引入了由模态演化差异（MED）引导的模态特异性时间尺度（MSTS）以及时域压缩LIF（TC-LIF）机制，将时间步从T=L-1压缩至T=log2(L)-1。在四个代表性MLLM及多种多模态基准上的实验表明，SpikeMLLM在激进的时间步压缩（Tv/Tt=3/4）下仍能保持近乎无损的性能，在InternVL2-8B和Qwen2VL-72B上相对于FP16基线的平均差距分别仅为0.72%和1.19%。我们进一步针对脉冲驱动数据流开发了专用RTL加速器，在面向部署的协同设计设置下，相较于FP16 GPU基线，实现了9.06倍的吞吐量提升和25.8倍的能效提升，展示了算法-硬件协同设计在实现高效多模态智能方面的潜力。