Large Language Model (LLM) inference is bottlenecked by the capacity and bandwidth of GPU High-Bandwidth Memory (HBM). Recent proposals, such as High-Bandwidth Flash (HBF) and RoMe, offer higher capacity or bandwidth than HBM, but require a minimum access granularity of kilobytes. We show that these Large-Granularity Memory Systems (LGMS) can degrade LLM inference performance by up to an order of magnitude. The root cause is read amplification during tiled matrix multiplication, in which each memory request fetches far more data than the compute tile needs. Such waste stems from a fundamental mismatch between the two-dimensional nature of compute tiles and the one-dimensional memory layout, causing each request to spill well beyond tile boundaries. To eliminate read amplification in LGMS, we propose to use tile-major layout. Rather than storing data as a one-dimensional strip, tile-major layout reshapes each contiguous memory block into a two-dimensional rectangle, aligning memory granularity with tile boundaries. To ease the adoption of tile-major layout on GPUs, we propose TileLens, lightweight software and hardware extensions that cover major classes of GPU kernels. TileLens-SW extends GPU DSLs so that DSL-based kernels can adopt tile-major by only changing the layout descriptor. TileLens-HW extends the Tensor Memory Accelerator (TMA) for tile-major support in TMA-based kernels without code changes. We evaluate TileLens on a cycle-level GPU simulator using Qwen-3 30B and Llama-3.1 70B. Combining a tile-major layout with our adaptive degree prefetcher, TileLens achieves near-HBM performance on HBF-augmented GPUs with a 5 us NAND read latency, reducing the geomean slowdown from 1.62-6.69x with conventional layouts to within 10% of an HBM-only baseline.

翻译：暂无翻译