Huge amounts of data being generated continuously by digitally interconnected systems of humans, organizations and machines. Data comes in variety of formats including structured, unstructured and semi-structured, what makes it impossible to apply the same standard approaches, techniques and algorithms to manage and process this data. Fortunately, the enterprise level distributed platform named Hadoop Ecosystem exists. This paper explores Apache Hive component that provides full stack data managements functionality in terms of Data Definition, Data Manipulation and Data Processing. Hive is a data warehouse system, which works with structured data stored in tables. Since, Hive works on top the Hadoop HDSFS, it benefits from extraordinary feature of HDFS including Fault Tolerance, Reliability, High Availability, Scalability, etc. In addition, Hive can take advantage of distributed computing power of the cluster through assigning jobs to MapReduce, Tez and Spark engines to run complex queries. The paper is focused on studying of Hive Data Model and analysis of processing performance done by MapReduce and Tez.
翻译:由人类、组织及机器组成的数字互联系统持续生成海量数据。这些数据涵盖结构化、非结构化及半结构化等多种格式,使得采用统一标准方法、技术与算法进行管理及处理成为不可能。所幸企业级分布式平台Hadoop生态系统应运而生。本文探索了Apache Hive组件,该组件在数据定义、数据操作与数据处理层面提供全栈数据管理功能。Hive作为数据仓库系统,可处理存储于表中的结构化数据。由于Hive基于Hadoop HDFS运行,它继承了HDFS的容错性、可靠性、高可用性及可扩展性等卓越特性。此外,Hive可通过将任务分配给MapReduce、Tez及Spark引擎来执行复杂查询,从而充分利用集群的分布式计算能力。本文重点研究Hive数据模型,并分析MapReduce与Tez的处理性能。