Modern applications commonly need to manage dataset types composed of heterogeneous data and schemas, making it difficult to access them in an integrated way. A single data store to manage heterogeneous data using a common data model is not effective in such a scenario, which results in the domain data being fragmented in the data stores that best fit their storage and access requirements (e.g., NoSQL, relational DBMS, or HDFS). Besides, organization workflows independently consume these fragments, and usually, there is no explicit link among the fragments that would be useful to support an integrated view. The research challenge tackled by this work is to provide the means to query heterogeneous data residing on distinct data repositories that are not explicitly connected. We propose a federated database architecture by providing a single abstract global conceptual schema to users, allowing them to write their queries, encapsulating data heterogeneity, location, and linkage by employing: (i) meta-models to represent the global conceptual schema, the remote data local conceptual schemas, and mappings among them; (ii) provenance to create explicit links among the consumed and generated data residing in separate datasets. We evaluated the architecture through its implementation as a polystore service, following a microservice architecture approach, in a scenario that simulates a real case in Oil \& Gas industry. Also, we compared the proposed architecture to a relational multidatabase system based on foreign data wrappers, measuring the user's cognitive load to write a query (or query complexity) and the query processing time. The results demonstrated that the proposed architecture allows query writing two times less complex than the one written for the relational multidatabase system, adding an excess of no more than 30% in query processing time.
翻译:现代应用通常需要管理由异构数据和模式组成的数据集类型,这使得以集成方式访问它们变得困难。在这种场景下,使用通用数据模型的单一数据存储来管理异构数据效果不佳,导致领域数据分散在最符合其存储和访问要求的数据存储中(例如,NoSQL、关系型数据库管理系统或HDFS)。此外,组织工作流独立地消费这些碎片,通常碎片之间缺乏有助于支持集成视图的显式链接。本研究应对的研究挑战是如何提供查询驻留在未显式连接的不同存储库上的异构数据的手段。我们提出了一种联邦数据库架构,通过向用户提供单一的抽象全局概念模式,允许他们编写查询,并通过以下方法封装数据异构性、位置和链接:(i) 元模型来表示全局概念模式、远程数据的局部概念模式以及它们之间的映射;(ii) 溯源来在消费和生成的、驻留在独立数据集中的数据之间创建显式链接。我们通过将其实现为一种遵循微服务架构方法的多存储服务,在模拟石油天然气行业真实案例的场景中评估了该架构。同时,我们将所提架构与基于外部数据包装器的关系型多数据库系统进行了比较,测量了用户编写查询的认知负荷(或查询复杂度)以及查询处理时间。结果表明,所提架构允许查询编写的复杂度仅为关系型多数据库系统的三分之一,而查询处理时间增加不超过30%。