Modern large language models (LLMs) increasingly depends on efficient long-context processing and generation mechanisms, including sparse attention, retrieval-augmented generation (RAG), and compressed contextual memory, to support complex reasoning. We show that these optimizations can be unified into a four-step memory processing pipeline: Prepare Memory, Compute Relevancy, Retrieval, and Apply to Inference. Through systematic profiling, we identify a 22%-97% memory processing overhead in LLM inference and strong heterogeneity in its computational characteristics. Motivated by this insight, we argue that \textbf{heterogeneous systems} are well-suited to accelerate memory processing and thus end-to-end inference. We demonstrate this approach on a GPU-FPGA system by offloading sparse, irregular, and memory-bounded operations to FPGAs while retaining compute-intensive operations on GPUs. Evaluated on an AMD MI210 GPU and an Alveo U55C FPGA, our system is up to $2.2\times$ faster and achieves up to $4.7\times$ less energy across multiple LLM inference optimizations than the GPU baseline (similar results hold on NVIDIA A100). These results establish heterogeneous systems as a practical direction for efficient LLM memory processing and inform future heterogeneous hardware design.

翻译：现代大语言模型（LLMs）日益依赖高效的长上下文处理与生成机制，包括稀疏注意力、检索增强生成（RAG）及压缩上下文记忆，以支持复杂推理。我们证明这些优化可统一为四步内存处理管线：准备内存、计算相关性、检索及应用于推理。通过系统性分析，我们发现LLM推理中存在22%-97%的内存处理开销，且其计算特征呈现高度异构性。基于这一洞察，我们提出异构系统非常适合加速内存处理，从而加速端到端推理。我们在GPU-FPGA系统上验证了该方法：将稀疏、不规则且受内存限制的操作卸载至FPGA，同时将计算密集型操作保留在GPU上。基于AMD MI210 GPU与Alveo U55C FPGA的评估表明，与GPU基线（NVIDIA A100上结果类似）相比，我们的系统在多种LLM推理优化场景下速度提升最高达2.2倍，能耗降低最高达4.7倍。这些结果确立了异构系统作为高效LLM内存处理的实用方向，并为未来异构硬件设计提供参考。