Preceding pen down the article, might want to stretch out appreciation to all the wellbeing teams beginning from cleaning/sterile group to Nurses, Doctors and other who are consistently battling to spare the mankind from continuous Covid-19 pandemic over the globe.

在下一篇文章之前，不妨向从清洁/无菌小组到护士，医生和其他一直在努力使人类免受全球Covid-19大流行的困扰的所有健康团队表示感谢。

The fundamental target of this article is to feature how we can load or import data into Hive tables without explicitly execute the “load” command. Basically, with this approach Data scientists can query or even visualize directly on various data visualization tool for quick investigation in a scenario when raw data is continuously ingested to HDFS based Data lake from the external sources on a consistent schedule. Otherwise, “load” command would be required to execute furthermore for stacking the processed data into Hive’s table. Here we are considering an existing environment with the following components either set up on the Cloud or on-premise.

本文的基本目标是介绍如何在不显式执行“ load”命令的情况下将数据加载或导入到Hive表中。 基本上，使用这种方法，当原始数据以一致的时间表从外部源连续摄取到基于HDFS的Data Lake时，数据科学家可以直接在各种数据可视化工具上进行查询甚至可视化，以进行快速调查。 否则，将需要“ load”命令来进一步执行，以将处理后的数据堆叠到Hive的表中。 在这里，我们正在考虑具有以下组件的现有环境，这些组件在云端或本地设置。

• Multi-node Cluster where HDFS installed and configured. Hive running on top of HDFS with MySQL database as metastore.
  已安装和配置HDFS的多节点群集。 Hive在HDFS之上运行，并将MySQL数据库作为metastore。
• Assuming raw data is getting dumped from multiple sources into HDFS Data lake landing zone by leveraging Kafka, Flume, customized data ingesting tool etc.
  假设利用Kafka，Flume，定制数据提取工具等将原始数据从多个来源转储到HDFS Data Lake登陆区。
• From the landing zone, raw data moves to the refining zone in order to clean junk and subsequently into the processing zone where clean data gets processed. Here we are considering that the processed data stored in text files with CSV format.
  原始数据从着陆区移至精炼区，以清理垃圾，然后移至处理区，在此处理干净数据。 在这里，我们考虑将处理后的数据存储在CSV格式的文本文件中。

Hive input is directory-based which similar to many Hadoop tools. This means, input for an operation is taken as files in a given directory. Using HDFS command, let’s create a directory in the HDFS using “$ hdfs dfs -mkdir <<name of the folder>>. Same can be done using Hadoop administrative UI depending upon user’s HDFS ACL settings. Now move the data files from the processing zone into newly created HDFS folder. As an example, here we are considering simple order data that ingested into the data lake and eventually transformed to consolidated text files with CSV format after cleaning and filtering. Few lines of rows are as follows

Hive输入是基于目录的，类似于许多Hadoop工具。 这意味着，操作的输入将作为给定目录中的文件。 使用HDFS命令，让我们使用“ $ hdfs dfs -mkdir <<文件夹名称>>在HDFS中创建一个目录。 根据用户的HDFS ACL设置，可以使用Hadoop管理UI进行相同的操作。 现在，将数据文件从处理区域移到新创建的HDFS文件夹中。 例如，这里我们考虑的是简单的订单数据，这些数据被导入到数据湖中，并在清洗和过滤后最终转换为CSV格式的合并文本文件。 行的几行如下

Next step is to create an external table in Hive by using the following command where the location is the path of HDFS directory that created on the previous step. here is the command we could use to create the external table using Hive CLI. The LOCATION statement in the command tells Hive where to find the input files.

下一步是使用以下命令在Hive中创建外部表，其中位置是在上一步中创建的HDFS目录的路径。 这是我们可以用来使用Hive CLI创建外部表的命令。 命令中的LOCATION语句告诉Hive在哪里找到输入文件。

If the command worked, an OK will be printed and upon executing Hive query, Hive engine fetches the data internally from these input text files by leveraging processing engine Map Reducer or other like Spark, Tez etc. Ideally, Spark or Tez can be configured as a processing engine in hive-site.xml in order to improve the data processing speed for a huge volume of input files.

如果该命令有效，则将打印OK，并且在执行Hive查询时，Hive引擎可利用处理引擎Map Reducer或其他诸如Spark，Tez等从这些输入文本文件内部获取数据。理想情况下，Spark或Tez可配置为hive-site.xml中的处理引擎，以提高大量输入文件的数据处理速度。

Once the table creation is successful, we can cross-check it on “ metastore” schema in the MySQL database. To perform that, log in to MySQL CLI which might be running on a different node in the cluster and then connect to the “metastore” database as well as pulls records from “TBLS” table. This displays the created Hive table information.

一旦表创建成功，我们就可以在MySQL数据库的“ metastore”模式中对其进行交叉检查。 要执行此操作，请登录到可能正在集群中其他节点上运行MySQL CLI，然后连接到“元存储”数据库并从“ TBLS”表中提取记录。 这将显示创建的Hive表信息。

The import can be verified through the Hive’s CLI by listing the first few rows in the table.

可以通过Hive的CLI列出表中的前几行来验证导入。

hive> Select * from OrderData;

蜂巢>从OrderData中选择*;

Additionally, “ analyze compute statistics “ command could be executed in Hive CLI to view the detail information of jobs that runs on that table.

另外，可以在Hive CLI中执行“ 分析计算统计信息 ”命令，以查看在该表上运行的作业的详细信息。

The primary advantage with this approach is, data can be query, analyze etc within a minimum span of time without additionally perform explicit data loading operation. Also helps the Data scientists to check the quality of data before running their machine learning jobs on the data lake or cluster. You could read here how to install and configure Apache Hive on multi-node Hadoop cluster with MySQL as Metastore.

这种方法的主要优点是，可以在最短的时间范围内查询，分析数据，而无需另外执行显式的数据加载操作。 还可以帮助数据科学家在数据湖或集群上运行其机器学习作业之前检查数据质量。 您可以在此处阅读如何在以MySQL作为Metastore的多节点Hadoop集群上安装和配置Apache Hive。

Written byGautam Goswami

由 Gautam Goswami 撰写

Enthusiastic about learning and sharing knowledge on Big Data and related headways. Play at the intersection of innovation, music and workmanship.

热衷于学习和共享有关大数据和相关进展的知识。 在创新，音乐和Craft.io的交汇处演奏。

Originally published at https://dataview.in on August 4, 2020.

最初于 2020年8月4日 在 https://dataview.in 上 发布 。