The past year has witnessed the increasing popularity of Large Language Models (LLMs). Their unprecedented scale and associated high hardware cost have impeded their broader adoption, calling for efficient hardware designs. With the large hardware needed to simply run LLM inference, evaluating different hardware designs becomes a new bottleneck. This work introduces LLMCompass, a hardware evaluation framework for LLM inference workloads. LLMCompass is fast, accurate, versatile, and able to describe and evaluate different hardware designs. LLMCompass includes a mapper to automatically find performance-optimal mapping and scheduling. It also incorporates an area-based cost model to help architects reason about their design choices. Compared to real-world hardware, LLMCompass' estimated latency achieves an average 10.4% error rate across various operators with various input sizes and an average 4.1% error rate for LLM inference. With LLMCompass, simulating a 4-NVIDIA A100 GPU node running GPT-3 175B inference can be done within 16 minutes on commodity hardware, including 26,400 rounds of the mapper's parameter search. With the aid of LLMCompass, this work draws architectural implications and explores new cost-effective hardware designs. By reducing the compute capability or replacing High Bandwidth Memory (HBM) with traditional DRAM, these new designs can achieve as much as 3.41x improvement in performance/cost compared to an NVIDIA A100, making them promising choices for democratizing LLMs. LLMCompass is planned to be fully open-source.

翻译：过去一年见证了大型语言模型（LLM）的日益普及。其前所未有的规模及高昂的硬件成本阻碍了更广泛的应用，亟需高效的硬件设计。由于运行LLM推理本身就需要大量硬件，评估不同硬件设计成为新的瓶颈。本文提出LLMCompass，一个面向LLM推理工作负载的硬件评估框架。LLMCompass具有快速、准确、通用的特点，能够描述和评估不同的硬件设计。该框架包含一个映射器，可自动寻找性能最优的映射与调度方案，并集成了基于面积的成本模型，帮助架构师审视其设计选择。与真实硬件相比，LLMCompass对不同输入规模下各类算子的延迟估计平均误差率为10.4%，对LLM推理的平均误差率为4.1%。借助LLMCompass，在通用硬件上模拟一个4块NVIDIA A100 GPU节点运行GPT-3 175B推理可在16分钟内完成，其中包括映射器的26,400轮参数搜索。基于LLMCompass，本文得出了架构启示，并探索了新的成本效益硬件设计。通过降低算力或使用传统DRAM替代高带宽内存（HBM），这些新设计相比NVIDIA A100可实现高达3.41倍的性能/成本提升，成为推动LLM民主化的可行方案。LLMCompass计划完全开源。