Context: Distributed Stream Processing Frameworks (DSPFs) are popular tools for expressing real-time Big Data applications that have to handle enormous volumes of data in real time. These frameworks distribute their applications over a cluster in order to scale horizontally along with the amount of incoming data. Inquiry: Crucial for the performance of such applications is the **distribution strategy** that is used to partition data and computations over the cluster nodes. In some DSPFs, like Apache Spark or Flink, the distribution strategy is hardwired into the framework which can lead to inefficient applications. The other end of the spectrum is offered by Apache Storm, which offers a low-level model wherein programmers can implement their own distribution strategies on a per-application basis to improve efficiency. However, this model conflates distribution and data processing logic, making it difficult to modify either. As a consequence, today's cluster application developers either have to accept the built-in distribution strategies of a high-level framework or accept the complexity of expressing a distribution strategy in Storm's low-level model. Approach: We propose a novel programming model wherein data processing operations and their distribution strategies are decoupled from one another and where new strategies can be created in a modular fashion. Knowledge: The introduced language abstractions cleanly separate the data processing and distribution logic of a stream processing application. This enables the expression of stream processing applications in a high-level framework while still retaining the flexibility offered by Storm's low-level model. Grounding: We implement our programming model as a domain-specific language, called Skitter, and use it to evaluate our approach. Our evaluation shows that Skitter enables the implementation of existing distribution strategies from the state of the art in a modular fashion. Our performance evaluation shows that the strategies implemented in Skitter exhibit the expected performance characteristics and that applications written in Skitter obtain throughput rates in the same order of magnitude as Storm. Importance: Our work enables developers to select the most performant distribution strategy for each operation in their application, while still retaining the programming model offered by high-level frameworks.

翻译：背景：分布式流处理框架（DSPF）是表达实时大数据应用的常用工具，这类应用需要实时处理海量数据。这些框架将应用程序分布在集群上，以便能够随输入数据量的增长而水平扩展。核心问题：此类应用性能的关键在于**分发策略**，即如何在集群节点间划分数据与计算。在某些DSPF（如Apache Spark或Flink）中，分发策略被固化在框架内部，这可能导致应用效率低下。另一极端是Apache Storm提供的底层模型，程序员可为每个应用自行实现分发策略以提升效率。然而，该模型将分发逻辑与数据处理逻辑混为一体，使得修改任一逻辑都变得困难。因此，当前的集群应用开发者要么必须接受高级框架内置的分发策略，要么必须承受在Storm底层模型中表达分发策略的复杂性。方法：我们提出一种新颖的编程模型，其中数据处理操作与其分发策略相互解耦，且新策略能以模块化方式创建。理论贡献：所引入的语言抽象清晰地分离了流处理应用的数据处理逻辑与分发逻辑。这使得流处理应用既能通过高级框架表达，同时仍保留Storm底层模型所提供的灵活性。实现验证：我们将该编程模型实现为一种名为Skitter的领域特定语言，并借此评估所提方法。评估表明，Skitter能够以模块化方式实现现有先进的分发策略。性能评估显示，Skitter实现的策略展现出预期的性能特征，且用Skitter编写的应用可获得与Storm同数量级的吞吐率。意义：本项工作使开发者能够为应用中的每个操作选择性能最优的分发策略，同时仍可沿用高级框架所提供的编程模型。