Deep Learning (DL) has developed to become a corner-stone in many everyday applications that we are now relying on. However, making sure that the DL model uses the underlying hardware efficiently takes a lot of effort. Knowledge about inference characteristics can help to find the right match so that enough resources are given to the model, but not too much. We have developed a DL Inference Performance Predictive Model (DIPPM) that predicts the inference latency, energy, and memory usage of a given input DL model on the NVIDIA A100 GPU. We also devised an algorithm to suggest the appropriate A100 Multi-Instance GPU profile from the output of DIPPM. We developed a methodology to convert DL models expressed in multiple frameworks to a generalized graph structure that is used in DIPPM. It means DIPPM can parse input DL models from various frameworks. Our DIPPM can be used not only helps to find suitable hardware configurations but also helps to perform rapid design-space exploration for the inference performance of a model. We constructed a graph multi-regression dataset consisting of 10,508 different DL models to train and evaluate the performance of DIPPM, and reached a resulting Mean Absolute Percentage Error (MAPE) as low as 1.9%.

翻译：深度学习已成为当今众多日常应用的核心技术。然而，确保深度学习模型高效利用底层硬件需要付出大量努力。了解推理特性有助于找到合适的资源匹配方案，既能保证模型获得足够资源，又不会过度分配。我们开发了深度学习推理性能预测模型（DIPPM），用于预测给定输入深度学习模型在NVIDIA A100 GPU上的推理延迟、能耗和内存使用情况。我们还设计了一种算法，可根据DIPPM的输出推荐合适的A100多实例GPU配置文件。我们提出了一种方法，可将以多种框架表达的深度学习模型转换为DIPPM所使用的通用图结构，这意味着DIPPM能够解析来自不同框架的输入深度学习模型。DIPPM不仅有助于寻找合适的硬件配置，还能对模型的推理性能进行快速设计空间探索。我们构建了一个包含10508个不同深度学习模型的图多回归数据集，用于训练和评估DIPPM的性能，最终平均绝对百分比误差低至1.9%。