The dynamic nature of resource allocation and runtime conditions on Cloud can result in high variability in a job's runtime across multiple iterations, leading to a poor experience. Identifying the sources of such variation and being able to predict and adjust for them is crucial to cloud service providers to design reliable data processing pipelines, provision and allocate resources, adjust pricing services, meet SLOs and debug performance hazards. In this paper, we analyze the runtime variation of millions of production SCOPE jobs on Cosmos, an exabyte-scale internal analytics platform at Microsoft. We propose an innovative 2-step approach to predict job runtime distribution by characterizing typical distribution shapes combined with a classification model with an average accuracy of >96%, out-performing traditional regression models and better capturing long tails. We examine factors such as job plan characteristics and inputs, resource allocation, physical cluster heterogeneity and utilization, and scheduling policies. To the best of our knowledge, this is the first study on predicting categories of runtime distributions for enterprise analytics workloads at scale. Furthermore, we examine how our methods can be used to analyze what-if scenarios, focusing on the impact of resource allocation, scheduling, and physical cluster provisioning decisions on a job's runtime consistency and predictability.

翻译：云环境中资源分配与运行时条件的动态特性会导致作业在多次迭代中的运行时间出现高度变异，从而带来不良体验。识别此类变异的根源并能够预测和调整它们，对于云服务提供商设计可靠的数据处理管道、配置和分配资源、调整定价服务、满足服务等级协议（SLO）以及调试性能瓶颈至关重要。本文分析了微软Cosmos平台（一个EB级内部分析平台）上数百万生产环境SCOPE作业的运行时变异。我们提出了一种创新的两步法，通过刻画典型的分布形态并结合分类模型来预测作业运行时分布，平均准确率超过96%，优于传统回归模型并能更准确地捕捉长尾分布。我们考察了作业计划特征与输入、资源分配、物理集群异构性与利用率以及调度策略等因素。据我们所知，这是首次针对企业级分析工作负载在规模上预测运行时分布类别的系统性研究。此外，我们探讨了如何利用所提方法分析假设场景，重点关注资源分配、调度和物理集群配置决策对作业运行时一致性和可预测性的影响。