Today, data analysis drives the decision-making process in virtually every human activity. This demands for software platforms that offer simple programming abstractions to express data analysis tasks and that can execute them in an efficient and scalable way. State-of-the-art solutions range from low-level programming primitives, which give control to the developer about communication and resource usage, but require significant effort to develop and optimize new algorithms, to high-level platforms that hide most of the complexities of parallel and distributed processing, but often at the cost of reduced efficiency. To reconcile these requirements, we developed Noir, a novel distributed data processing platform written in Rust. Noir provides a high-level dataflow programming model as mainstream data processing systems. It supports static and streaming data, it enables data transformations, grouping, aggregation, iterative computations, and time-based analytics, incurring in a low overhead. This paper presents In this paper, we present the programming model and the implementation details of Noir. We evaluate it under heterogeneous workloads. We compare it with state-of-the-art solutions for data analysis and high-performance computing, as well as alternative research products, which offer different programming abstractions and implementation strategies. Noir programs are compact and easy to write: developers need not care about low-level concerns such as resource usage, data serialization, concurrency control, and communication. Noir consistently presents comparable or better performance than competing solutions, by a large margin in several scenarios. We conclude that Noir offers a good tradeoff between simplicity and performance, allowing developers to easily express complex data analysis tasks and achieve high performance and scalability.

翻译：如今，数据分析驱动着几乎所有人类活动中的决策过程。这要求软件平台既能提供简单的编程抽象以表达数据分析任务，又能以高效且可扩展的方式执行这些任务。现有解决方案涵盖了从低层级编程原语（赋予开发者对通信和资源使用的控制权，但开发和优化新算法需要大量精力）到高层级平台（隐藏了并行和分布式处理的大部分复杂性，但通常以降低效率为代价）的多种选择。为了调和这些需求，我们开发了Noir——一个用Rust编写的新型分布式数据处理平台。Noir提供了与主流数据处理系统类似的高层级数据流编程模型。它支持静态数据和流式数据，能够实现数据转换、分组、聚合、迭代计算以及基于时间的分析，且开销较低。本文介绍了Noir的编程模型和实现细节。我们在异构工作负载下对其进行了评估，并与用于数据分析和高性能计算的现有解决方案以及其他提供不同编程抽象和实现策略的研究产品进行了比较。Noir程序紧凑且易于编写：开发者无需关心资源使用、数据序列化、并发控制和通信等底层问题。Noir的性能始终与竞品相当或更优，在多个场景中大幅领先。我们得出结论：Noir在简洁性与性能之间实现了良好平衡，使开发者能够轻松表达复杂的数据分析任务，同时实现高性能和可扩展性。