While providing low latency is a fundamental requirement in deploying recommendation services, achieving high resource utility is also crucial in cost-effectively maintaining the datacenter. Co-locating multiple workers of a model is an effective way to maximize query-level parallelism and server throughput, but the interference caused by concurrent workers at shared resources can prevent server queries from meeting its SLA. Hera utilizes the heterogeneous memory requirement of multi-tenant recommendation models to intelligently determine a productive set of co-located models and its resource allocation, providing fast response time while achieving high throughput. We show that Hera achieves an average 37.3% improvement in effective machine utilization, enabling 26% reduction in required servers, significantly improving upon the baseline recommedation inference server.

翻译：低延迟是部署推荐服务的基本要求，而实现高资源利用率对于经济高效地维护数据中心同样至关重要。将同一模型的多个工作线程共置是最大化查询级别并行度和服务器吞吐量的有效方式，但共享资源上并发工作线程引发的干扰可能使服务器查询无法满足服务等级协议（SLA）。Hera利用多租户推荐模型的异构内存需求，智能地确定一组高效的共置模型及其资源分配方案，在实现高吞吐量的同时提供快速响应时间。实验表明，Hera平均有效机器利用率提升37.3%，所需服务器数量减少26%，与基线推荐推理服务器相比实现了显著改进。