In this work, we present a new benchmarking suite with new real-life inspired skewed workloads to test the performance of concurrent index data structures. We started this project to prepare workloads specifically for self-adjusting data structures, i.e., they handle more frequent requests faster, and, thus, should perform better than their standard counterparts. We looked over the commonly used suites to test performance of concurrent indices trying to find an inspiration: Synchrobench, Setbench, YCSB, and TPC - and we found several issues with them. The major problem is that they are not flexible: it is difficult to introduce new workloads, it is difficult to set the duration of the experiments, and it is difficult to change the parameters. We decided to solve this issue by presenting a new suite based on Synchrobench. Finally, we highlight the problem of measuring performance of data structures. We show that the relative performance of data structures highly depends on the workload: it is not clear which data structure is best. For that, we take three state-of-the-art concurrent binary search trees and run them on the workloads from our benchmarking suite. As a result, we get six experiments with all possible relative performance of the chosen data structures.

翻译：本文提出了一种新的基准测试套件，采用受真实场景启发的倾斜工作负载，用于评估并发索引数据结构的性能。我们启动该项目旨在为自适应数据结构准备特定工作负载——这类结构能更快处理高频请求，因此理论上应优于标准对应结构。通过调研当前常用的并发索引性能测试套件（Synchrobench、Setbench、YCSB 和 TPC），我们发现若干问题：主要缺陷在于灵活性不足——难以引入新工作负载、难以设定实验时长、难以调整参数。为此，我们基于 Synchrobench 开发了新套件以解决这些问题。最后，我们揭示了数据结构性能评估的难点：研究表明，数据结构的相对性能高度依赖于具体工作负载，无法明确判定最优方案。通过选取三种最先进的并发二叉搜索树，并在我们的基准套件工作负载上进行测试，最终获得了涵盖所有相对性能可能性的六组实验结果。