Cloud computing and virtualization solutions allow one to rent the virtual machines (VMs) needed to run applications on a pay-per-use basis, but rented VMs do not offer any guarantee on their performance. Cloud platforms are known to be affected by performance variability, but a better understanding is still required. This paper moves in that direction and presents an in-depth, multi-faceted study on the performance variability of VMs. Unlike previous studies, our assessment covers a wide range of factors: 16 VM types from 4 well-known cloud providers, 10 benchmarks, and 28 different metrics. We present four new contributions. First, we introduce a new benchmark suite (VMBS) that let researchers and practitioners systematically collect a diverse set of performance data. Second, we present a new indicator, called Variability Indicator, that allows for measuring variability in the performance of VMs. Third, we illustrate an analysis of the collected data across four different dimensions: resources, isolation, time, and cost. Fourth, we present multiple predictive models based on Machine Learning that aim to forecast future performance and detect time patterns. Our experiments provide important insights on the resource variability of VMs, highlighting differences and similarities between various cloud providers. To the best of our knowledge, this is the widest analysis ever conducted on the topic.

翻译：云计算与虚拟化解决方案允许用户按需租用运行应用程序所需的虚拟机（VM），但所租用的虚拟机并不保证其性能表现。云平台已知会受到性能变异的影响，但仍需更深入的理解。本文旨在推动该方向的研究，并呈现对虚拟机性能变异的深入且多维度分析。与以往研究不同，我们的评估涵盖广泛因素：来自4家知名云提供商的16种虚拟机类型、10个基准测试以及28项不同指标。我们提出四项新贡献。首先，引入一套新型基准测试套件（VMBS），使研究人员和从业者能够系统性地收集多样化的性能数据。其次，提出一项名为变异指标（Variability Indicator）的新指标，用于量化虚拟机性能的变异程度。第三，从资源、隔离、时间和成本四个维度对收集的数据进行分析。第四，基于机器学习构建多种预测模型，旨在预测未来性能并检测时间模式。我们的实验揭示了虚拟机资源变异的关键见解，突出了不同云提供商之间的差异与共性。据我们所知，这是该领域迄今为止规模最广泛的分析研究。