Meeting service-level objectives (SLOs) in Large Language Models (LLMs) serving is critical, but managing the high variability in load presents a significant challenge. Recent advancements in FP8 inference, backed by native hardware support, offer a potential solution: executing FP16 models by default, while switching to FP8 models during sudden load surges to achieve higher throughput at the cost of a slight quality degradation. Although this approach facilitates effective SLO management, it introduces additional memory overhead due to storing two versions of the same model. In response, this paper proposes NestedFP, an LLM serving technique that supports both FP16 and FP8 models in a memory-efficient manner by overlaying FP8 parameters onto FP16 parameters, allowing both models to share the same FP16 memory footprint. By leveraging a compact data format for the overlay and a specialized GEMM kernel optimized for this format, NestedFP ensures minimal degradation in both model quality and inference throughput across both FP8 and FP16 modes. NestedFP provides a flexible platform for dynamic, SLO-aware precision selection. The code is available at https://github.com/SNU-ARC/NestedFP.

翻译：在大语言模型（LLM）服务中满足服务水平目标（SLO）至关重要，但负载的高可变性带来了重大挑战。近期在FP8推理方面的进展，凭借其原生硬件支持，提供了一种潜在的解决方案：默认执行FP16模型，同时在突发负载激增期间切换至FP8模型，以轻微的质量下降为代价实现更高的吞吐量。尽管这种方法有助于实现有效的SLO管理，但由于需要存储同一模型的两个版本，会引入额外的内存开销。为此，本文提出NestedFP，一种LLM服务技术，通过将FP8参数叠加到FP16参数之上，以内存高效的方式同时支持FP16和FP8模型，使两个模型共享相同的FP16内存占用。通过利用一种紧凑的叠加数据格式以及针对该格式优化的专用GEMM内核，NestedFP确保了在FP8和FP16两种模式下，模型质量和推理吞吐量的下降都降至最低。NestedFP为动态、SLO感知的精度选择提供了一个灵活的平台。代码可在 https://github.com/SNU-ARC/NestedFP 获取。