Modern transformer-based Large Language Models (LLMs) are constructed with a series of decoder blocks. Each block comprises three key components: (1) QKV generation, (2) multi-head attention, and (3) feed-forward networks. In batched processing, QKV generation and feed-forward networks involve compute-intensive matrix-matrix multiplications (GEMM), while multi-head attention requires bandwidth-heavy matrix-vector multiplications (GEMV). Machine learning accelerators like TPUs or NPUs are proficient in handling GEMM but are less efficient for GEMV computations. Conversely, Processing-in-Memory (PIM) technology is tailored for efficient GEMV computation, while it lacks the computational power to handle GEMM effectively. Inspired by this insight, we propose NeuPIMs, a heterogeneous acceleration system that jointly exploits a conventional GEMM-focused NPU and GEMV-optimized PIM devices. The main challenge in efficiently integrating NPU and PIM lies in enabling concurrent operations on both platforms, each addressing a specific kernel type. First, existing PIMs typically operate in a "blocked" mode, allowing only either NPU or PIM to be active at any given time. Second, the inherent dependencies between GEMM and GEMV in LLMs restrict their parallel processing. To tackle these challenges, NeuPIMs is equipped with dual row buffers in each bank, facilitating the simultaneous management of memory read/write operations and PIM commands. Further, NeuPIMs employs a runtime sub-batch interleaving technique to maximize concurrent execution, leveraging batch parallelism to allow two independent sub-batches to be pipelined within a single NeuPIMs device. Our evaluation demonstrates that compared to GPU-only, NPU-only, and a na\"ive NPU+PIM integrated acceleration approaches, NeuPIMs achieves 3$\times$, 2.4$\times$ and 1.6$\times$ throughput improvement, respectively.

翻译：现代基于Transformer的大语言模型由一系列解码器模块构成。每个模块包含三个关键组件：（1）QKV生成、（2）多头注意力和（3）前馈网络。在批量处理中，QKV生成和前馈网络涉及计算密集型的矩阵-矩阵乘法（GEMM），而多头注意力则需要带宽密集型的矩阵-向量乘法（GEMV）。TPU或NPU等机器学习加速器擅长处理GEMM，但GEMV计算效率较低。相反，存内处理技术专为高效GEMV计算而设计，却缺乏有效处理GEMM的计算能力。受此启发，我们提出NeuPIMs——一种异构加速系统，协同利用专注于GEMM的传统NPU和针对GEMV优化的PIM设备。高效集成NPU与PIM的主要挑战在于实现两种平台（各自处理特定核类型）的并发运算。首先，现有PIM通常以"阻塞"模式运行，任意时刻仅允许NPU或PIM单一设备激活。其次，LLM中GEMM与GEMV的固有依赖关系限制了它们的并行处理。为应对这些挑战，NeuPIMs在每个存储体中配备双行缓冲器，可同时管理内存读写操作和PIM指令。进一步地，NeuPIMs采用运行时子批次交错技术最大化并发执行，通过利用批次并行性，使两个独立子批次可在单个NeuPIMs设备内流水线化。实验评估表明，与纯GPU、纯NPU以及朴素NPU+PIM集成加速方案相比，NeuPIMs的吞吐量分别提升3倍、2.4倍和1.6倍。