The rapid advancement of big data technologies has underscored the need for robust and efficient data processing solutions. Traditional Spark-based Platform-as-a-Service (PaaS) solutions, such as Databricks and Amazon Web Services Elastic MapReduce, provide powerful analytics capabilities but often result in high operational costs and vendor lock-in issues. These platforms, while user-friendly, can lead to significant inefficiencies due to their cost structures and lack of transparent pricing. This paper introduces a cost-effective and flexible orchestration framework using Dagster. Our solution aims to reduce dependency on any single PaaS provider by integrating various Spark execution environments. We demonstrate how Dagster's orchestration capabilities can enhance data processing efficiency, enforce best coding practices, and significantly reduce operational costs. In our implementation, we achieved a 12% performance improvement over EMR and a 40% cost reduction compared to DBR, translating to over 300 euros saved per pipeline run. Our goal is to provide a flexible, developer-controlled computing environment that maintains or improves performance and scalability while mitigating the risks associated with vendor lock-in. The proposed framework supports rapid prototyping and testing, which is essential for continuous development and operational efficiency, contributing to a more sustainable model of large data processing.

翻译：大数据技术的快速发展凸显了对稳健高效数据处理解决方案的需求。传统的基于Spark的平台即服务（PaaS）解决方案，如Databricks和Amazon Web Services Elastic MapReduce，提供了强大的分析能力，但往往导致高昂的运营成本和供应商锁定问题。这些平台虽然用户友好，但由于其成本结构和缺乏透明的定价，可能导致显著的效率低下。本文介绍了一种使用Dagster的经济高效且灵活的编排框架。我们的解决方案旨在通过集成各种Spark执行环境来减少对任何单一PaaS提供商的依赖。我们展示了Dagster的编排能力如何提高数据处理效率、强制执行最佳编码实践并显著降低运营成本。在我们的实施中，相比EMR实现了12%的性能提升，与DBR相比成本降低了40%，相当于每次流水线运行节省超过300欧元。我们的目标是提供一个灵活的、由开发者控制的计算环境，该环境保持或提升性能和可扩展性，同时减轻与供应商锁定相关的风险。所提出的框架支持快速原型设计和测试，这对于持续开发和运营效率至关重要，有助于实现更可持续的大数据处理模式。