The evolution of data architecture has seen the rise of data lakes, aiming to solve the bottlenecks of data management and promote intelligent decision-making. However, this centralized architecture is limited by the proliferation of data sources and the growing demand for timely analysis and processing. A new data paradigm, Data Mesh, is proposed to overcome these challenges. Data Mesh treats domains as a first-class concern by distributing the data ownership from the central team to each data domain, while keeping the federated governance to monitor domains and their data products. Many multi-million dollar organizations like Paypal, Netflix, and Zalando have already transformed their data analysis pipelines based on this new architecture. In this decentralized architecture where data is locally preserved by each domain team, traditional centralized machine learning is incapable of conducting effective analysis across multiple domains, especially for security-sensitive organizations. To this end, we introduce a pioneering approach that incorporates Federated Learning into Data Mesh. To the best of our knowledge, this is the first open-source applied work that represents a critical advancement toward the integration of federated learning methods into the Data Mesh paradigm, underscoring the promising prospects for privacy-preserving and decentralized data analysis strategies within Data Mesh architecture.

翻译：数据架构的演进见证了数据湖的兴起，旨在解决数据管理瓶颈并促进智能决策。然而，这种集中式架构受到数据源激增以及对实时分析与处理日益增长需求的制约。为克服这些挑战，一种新型数据范式——数据网格被提出。数据网格将领域作为首要关注点，通过将数据所有权从中央团队分配给各个数据领域，同时保留联邦治理以监控领域及其数据产品。Paypal、Netflix、Zalando等众多市值数十亿美元的组织已基于该新架构重构其数据分析流水线。在此类数据由各领域团队本地保存的分布式架构中，传统集中式机器学习无法跨多个领域进行有效分析，尤其对于安全敏感型组织而言。为此，我们提出一种开创性方法，将联邦学习融入数据网格。据我们所知，这是首个开源应用工作，标志着向数据网格范式中整合联邦学习方法的关键进展，凸显了数据网格架构内隐私保护与去中心化数据分析策略的广阔前景。