Model parallelism is conventionally viewed as a method to scale a single large deep learning model beyond the memory limits of a single device. In this paper, we demonstrate that model parallelism can be additionally used for the statistical multiplexing of multiple devices when serving multiple models, even when a single model can fit into a single device. Our work reveals a fundamental trade-off between the overhead introduced by model parallelism and the opportunity to exploit statistical multiplexing to reduce serving latency in the presence of bursty workloads. We explore the new trade-off space and present a novel serving system, AlpaServe, that determines an efficient strategy for placing and parallelizing collections of large deep learning models across a distributed cluster. Evaluation results on production workloads show that AlpaServe can process requests at up to 10x higher rates or 6x more burstiness while staying within latency constraints for more than 99% of requests.

翻译：模型并行传统上被视为将单个大型深度学习模型扩展到超出单个设备内存限制的方法。本文表明，即使在单个模型可容纳于单个设备的情况下，模型并行还可用于服务多个模型时对多个设备进行统计复用。我们的研究揭示了模型并行引入的开销与利用统计复用在突发负载下降低服务延迟的机遇之间的基本权衡。我们探索了这一新的权衡空间，并提出了一套新型服务系统AlpaServe，该系统能够确定在分布式集群中放置和并行化大型深度学习模型集合的高效策略。基于生产工作负载的评估结果表明，AlpaServe在满足超过99%请求的延迟约束条件下，可处理高达10倍请求率或6倍突发性的负载。