In long-context large language model (LLM) inference, the prefill stage dominates computation due to self-attention over the complete input context. Sparse attention significantly reduces self-attention computation by limiting each token's interactions to a subset of tokens. The attention sparsity pattern varies across input prompts, and within a prompt, each attention head can follow a distinct pattern. This makes attention sparsity dynamic. The requirement of generating the sparsity pattern, combined with limited data reuse in attention, shifts the prefill compute to being memory-bound. This, in addition to the huge energy requirements for long-context inference on GPU, motivates FPGAs as good candidates for accelerating dynamic long-context inference. To tackle these challenges, we propose FAST-Prefill, the first FPGA accelerator for long-context prefill-stage inference with dynamic sparse attention. To efficiently generate sparse indices, we propose a \textit{fused pipeline unit with a memory-aware execution order} to reduce large tensors and irregular memory accesses. To reduce off-chip memory traffic for accessing the KV cache, we utilize the memory hierarchy to design a \textit{liveness-driven, dual-tier cache}. For high-throughput matrix multiplication, we design a \textit{hybrid Matrix Processing Unit (MPU)} with DSPs and bit-plane decomposition using LUTs. We implement FAST-Prefill on Alveo U280 and evaluate it on the Llama and Qwen models (batch size = 1) for context lengths ranging from 4K to 128K tokens. We demonstrate an average speedup of up to 2.5$\times$ in TTFT and 4.5$\times$ improvement in energy efficiency over GPU implementation on Nvidia A5000 GPU.

翻译：在长上下文大语言模型（LLM）推理中，由于需要对完整输入上下文进行自注意力计算，预填充阶段主导了整体计算开销。稀疏注意力通过将每个令牌的交互限制在令牌子集内，显著减少了自注意力计算量。注意力稀疏模式随输入提示的不同而变化，且在同一提示内，每个注意力头可遵循不同的模式，这使得注意力稀疏性呈现动态特性。生成稀疏模式的需求，加之注意力计算中有限的数据复用，使得预填充计算转变为内存受限型任务。此外，在GPU上进行长上下文推理的巨大能耗需求，促使FPGA成为加速动态长上下文推理的理想候选平台。为应对这些挑战，我们提出了FAST-Prefill——首个支持动态稀疏注意力的长上下文预填充阶段FPGA加速器。为高效生成稀疏索引，我们提出了一种\textit{融合流水线单元与内存感知执行顺序}，以减少大张量和不规则内存访问。为降低访问KV缓存的片外内存流量，我们利用内存层次结构设计了\textit{活性驱动的双层缓存}。针对高吞吐量矩阵乘法，我们设计了\textit{混合矩阵处理单元（MPU）}，结合DSP和基于LUT的位平面分解技术。我们在Alveo U280平台上实现了FAST-Prefill，并在Llama和Qwen模型（批大小=1）上对4K至128K令牌的上下文长度进行了评估。实验表明，相较于Nvidia A5000 GPU实现，本方案在TTFT上实现了最高2.5$\times$的平均加速，并在能效上获得4.5$\times$的提升。