Today, data analysis drives the decision-making process in virtually every human activity. This demands for software platforms that offer simple programming abstractions to express data analysis tasks and that can execute them in an efficient and scalable way. State-of-the-art solutions range from low-level programming primitives, which give control to the developer about communication and resource usage, but require significant effort to develop and optimize new algorithms, to high-level platforms that hide most of the complexities of parallel and distributed processing, but often at the cost of reduced efficiency. To reconcile these requirements, we developed Noir, a novel distributed data processing platform written in Rust. Noir provides a high-level dataflow programming model as mainstream data processing systems. It supports static and streaming data, it enables data transformations, grouping, aggregation, iterative computations, and time-based analytics, incurring in a low overhead. This paper presents In this paper, we present the programming model and the implementation details of Noir. We evaluate it under heterogeneous workloads. We compare it with state-of-the-art solutions for data analysis and high-performance computing, as well as alternative research products, which offer different programming abstractions and implementation strategies. Noir programs are compact and easy to write: developers need not care about low-level concerns such as resource usage, data serialization, concurrency control, and communication. Noir consistently presents comparable or better performance than competing solutions, by a large margin in several scenarios. We conclude that Noir offers a good tradeoff between simplicity and performance, allowing developers to easily express complex data analysis tasks and achieve high performance and scalability.

翻译：当今，数据分析已渗透至人类各项活动的决策过程。这要求软件平台既能提供简洁的编程抽象以表达数据分析任务，又能以高效可扩展的方式执行这些任务。现有解决方案涵盖两个极端：底层编程原语虽赋予开发者对通信与资源使用的控制权，却需要投入大量精力开发与优化新算法；高层平台虽隐藏了并行与分布式处理的复杂性，却往往牺牲了运行效率。为调和这些需求，我们开发了Noir——一款基于Rust语言的新型分布式数据处理平台。Noir提供与主流数据处理系统同等抽象层次的高阶数据流编程模型，支持静态与流式数据，实现数据转换、分组、聚合、迭代计算及时间序列分析，且运行时开销极低。本文详述了Noir的编程模型与实现细节，并通过异构负载对其性能进行评估。我们将其与数据分析、高性能计算领域的现有方案及替代性研究产品进行对比——这些产品采用不同的编程抽象与实现策略。实验表明，Noir程序简洁易写，开发者无需关注资源使用、数据序列化、并发控制及通信等底层细节。在多数场景下，Noir展现出与竞品相当甚至显著更优的性能表现。结论证实，Noir在简洁性与性能间实现了理想平衡，使开发者既能轻松构建复杂分析任务，又能获得高吞吐与可扩展性。