The use of machine learning (ML) inference for various applications is growing drastically. ML inference services engage with users directly, requiring fast and accurate responses. Moreover, these services face dynamic workloads of requests, imposing changes in their computing resources. Failing to right-size computing resources results in either latency service level objectives (SLOs) violations or wasted computing resources. Adapting to dynamic workloads considering all the pillars of accuracy, latency, and resource cost is challenging. In response to these challenges, we propose InfAdapter, which proactively selects a set of ML model variants with their resource allocations to meet latency SLO while maximizing an objective function composed of accuracy and cost. InfAdapter decreases SLO violation and costs up to 65% and 33%, respectively, compared to a popular industry autoscaler (Kubernetes Vertical Pod Autoscaler).

翻译：机器学习（ML）推理在各领域的应用正在急剧增长。ML推理服务直接与用户交互，需要快速且准确的响应。此外，这些服务面临动态的请求工作负载，导致其计算资源需求不断变化。若无法合理调整计算资源规模，将导致延迟服务等级目标（SLO）违规或计算资源浪费。在兼顾精度、延迟与资源成本三大基石的前提下适应动态工作负载极具挑战性。针对这些问题，我们提出InfAdapter方法，该方法通过主动选择一组ML模型变体及其资源配置，在满足延迟SLO的同时最大化由精度与成本构成的目标函数。与业界广泛应用的自动扩缩器（Kubernetes Vertical Pod Autoscaler）相比，InfAdapter将SLO违规率和成本分别降低了65%和33%。