State Space Models (SSMs) such as Mamba and Mamba-2 achieve linear-time autoregressive inference, making them attractive for latency-sensitive and resource-constrained deployment. Yet every generated token still reads the full input and output projection weights of every layer from off-chip memory, and on an FPGA these two projections take the majority of a layer's cycles. Existing accelerators lower this cost by quantizing the weights or by speeding up the scan, but none reduce the number of weight elements a token must read, and a low-rank factorization on its own does not help: without a datapath built for it, the two-stage product takes as long as the dense one. We present \textbf{LowRank-SSM}, a hardware-software co-design framework that jointly addresses three constraints of rank-reduced Mamba inference: where rank should be spent, how the resulting factors should execute, and how reduced weight traffic should be delivered by the FPGA memory system. A sensitivity-guided per-projection search allocates rank under a perplexity ceiling while restricting compression to Mamba-2 operands that tolerate approximation. A rank-configurable projection datapath maps both factor stages onto the existing GEMV engines, keeping the intermediate vector on chip and changing rank through loop bounds and weight images rather than bitstream recompilation. A platform-aware NoC/AXI memory subsystem then balances and prefetches weight streams to sustain 42\,GB/s. Together, these software and hardware choices convert structural rank reduction into measured latency reduction: in our ablation, 93\% of the projection traffic removed by rank reduction appears as saved token latency. With a rank schedule found under a 5\% perplexity ceiling, which costs 1.6 points of average zero-shot accuracy, the complete accelerator decodes Mamba2-2.7B at 16.79 tokens/s and 0.41\,J per token.

翻译：暂无翻译