The increasing size of language models necessitates a thorough analysis across multiple dimensions to assess trade-offs among crucial hardware metrics such as latency, energy consumption, GPU memory usage, and performance. Identifying optimal model configurations under specific hardware constraints is becoming essential but remains challenging due to the computational load of exhaustive training and evaluation on multiple devices. To address this, we introduce HW-GPT-Bench, a hardware-aware benchmark that utilizes surrogate predictions to approximate various hardware metrics across 13 devices of architectures in the GPT-2 family, with architectures containing up to 774M parameters. Our surrogates, via calibrated predictions and reliable uncertainty estimates, faithfully model the heteroscedastic noise inherent in the energy and latency measurements. To estimate perplexity, we employ weight-sharing techniques from Neural Architecture Search (NAS), inheriting pretrained weights from the largest GPT-2 model. Finally, we demonstrate the utility of HW-GPT-Bench by simulating optimization trajectories of various multi-objective optimization algorithms in just a few seconds.

翻译：语言模型规模的日益增长，要求我们从多个维度进行深入分析，以评估关键硬件指标（如延迟、能耗、GPU内存使用和性能）之间的权衡。在特定硬件约束下识别最优模型配置正变得至关重要，但由于在多设备上进行全面训练和评估的计算负载巨大，这仍然具有挑战性。为此，我们引入了HW-GPT-Bench，这是一个硬件感知基准，它利用代理预测来近似估算GPT-2系列中多达13种架构（参数规模高达7.74亿）在各种设备上的多种硬件指标。我们的代理模型通过校准预测和可靠的不确定性估计，忠实地模拟了能耗和延迟测量中固有的异方差噪声。为了估计困惑度，我们采用了来自神经架构搜索（NAS）的权重共享技术，继承了最大GPT-2模型的预训练权重。最后，我们通过模拟多种多目标优化算法在短短几秒钟内的优化轨迹，展示了HW-GPT-Bench的实用性。