Hive 完整部署指南

作者：🧑‍🚀 deadmau5v 发布于 2023/3/14

需要先完成 Hive最小化部署的操作并启动

Hadoop
MySQL

1 配置MySQL元数据

# 连接到mysql
mysql -u root -p

创建Hive元数据库

create database metastore;
exit;

复制驱动 JDBC 到 Hive lib目录

cp /opt/softs/mysql-connector-java-5.1.37.jar $HIVE_HOME/lib/

在Hive conf 目录下新建并编辑hive-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
      <property>
              <name>hive.metastore.warehouse.dir</name>
              <value>/user/hive/warehouse</value>
      </property>
      <property>
              <name>hive.metastore.local</name>
              <value>true</value>
      </property>
      <property>
              <name>javax.jdo.option.ConnectionURL</name>
              <value>jdbc:mysql://master:3306/metastore?createDatabaseIfNotExist=true</value>
      </property>
      <property>
              <name>javax.jdo.option.ConnectionDriverName</name>
              <value>com.mysql.jdbc.Driver</value>
      </property>
      <property>
              <name>javax.jdo.option.ConnectionUserName</name>
              <value>root</value>
      </property>
      <property>
              <name>javax.jdo.option.ConnectionPassword</name>
              <value>123456</value>
      </property>
</configuration>

初始化 hive

cd $HIVE_HOME
bin/schematool -dbType mysql -initSchema -verbose

HiveServer2 部署

启用 hadoop 代理用户修改所有机器配置文件添加以下配置项

<!--配置所有节点的 root 用户都可作为代理用户-->
<property>
  <name>hadoop.proxyuser.root.hosts</name>
  <value>*</value>
</property>

<!--配置 root 用户能够代理的用户组为任意组-->
<property>
  <name>hadoop.proxyuser.root.groups</name>
  <value>*</value>
</property>

<!--配置 root 用户能够代理的用户为任意用户-->
<property>
  <name>hadoop.proxyuser.root.users</name>
  <value>*</value>
</property>

修改 hive-site

<!-- 指定hiveserver2连接的host -->
<property>
	<name>hive.server2.thrift.bind.host</name>
	<value>0.0.0.0</value>
</property>

<!-- 指定hiveserver2连接的端口号 -->
<property>
	<name>hive.server2.thrift.port</name>
	<value>10000</value>
</property>

启动 metastore

$HIVE_HOME/bin/hive --service metastore

启动 hiveserver2

$HIVE_HOME/bin/hiveserver2
# 后台运行
nohup bin/hiveserver2 >/dev/null 2>&1 &

jps 查看

[root@master ~]# jps
1712 NameNode
3219 RunJar   # 👈 启动成功！
2549 NodeManager
1928 DataNode
3466 Jps

使用 beeline 连接测试

$HIVE_HOME/bin/beeline -u jdbc:hive2://master:10000 -n root

show databases;
show tables;

metastore 独立服务模式部署

修改 hive-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
        <property>
                <name>hive.metastore.warehouse.dir</name>
                <value>/user/hive/warehouse</value>
        </property>
   <!-- jdbc连接的URL -->
    <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://master:3306/metastore?useSSL=false</value>
    </property>

    <!-- jdbc连接的Driver-->
    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
    </property>

        <!-- jdbc连接的username-->
    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>root</value>
    </property>

    <!-- jdbc连接的password -->
    <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>123456</value>
    </property>

    <property>
        <name>hive.metastore.uris</name>
        <value>thrift://master:9083</value>
    </property>


<!-- 指定hiveserver2连接的host -->
<property>
        <name>hive.server2.thrift.bind.host</name>
        <value>0.0.0.0</value>
</property>

<!-- 指定hiveserver2连接的端口号 -->
<property>
        <name>hive.server2.thrift.port</name>
        <value>10000</value>
</property>
</configuration>

启动 metastore 服务

bin/hive --service metastore

测试运行

bin/hive
show databases;

部署完成！

报错合集

报错 #1 《配置文件错误》

2023-11-14 10:04:34,993 ERROR [main] conf.Configuration (Configuration.java:loadResource(3023)) - error parsing conf file:/opt/module/hive-3.1.2/conf/hive-site.xml
com.ctc.wstx.exc.WstxParsingException: Unexpected close tag </property>; expected </value>.
at [row,col,system-id]: [8,11,"file:/opt/module/hive-3.1.2/conf/hive-site.xml"]
at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:621)
at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:491)
at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:475)
at com.ctc.wstx.sr.BasicStreamReader.reportWrongEndElem(BasicStreamReader.java:3365)
at com.ctc.wstx.sr.BasicStreamReader.readEndElem(BasicStreamReader.java:3292)
at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2911)
at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1123)
at org.apache.hadoop.conf.Configuration$Parser.parseNext(Configuration.java:3320)
        at org.apache.hadoop.conf.Configuration$Parser.parse(Configuration.java:3114)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3007)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2973)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2848)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1460)
at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:4996)
at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:5069)
at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5156)
at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5104)
at org.apache.hive.beeline.HiveSchemaTool.<init>(HiveSchemaTool.java:96)
at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1473)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Exception in thread "main" java.lang.RuntimeException: com.ctc.wstx.exc.WstxParsingException: Unexpected close tag </property>; expected </value>.
at [row,col,system-id]: [8,11,"file:/opt/module/hive-3.1.2/conf/hive-site.xml"]
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3024)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2973)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2848)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1460)
at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:4996)
at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:5069)
at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5156)
at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5104)
at org.apache.hive.beeline.HiveSchemaTool.<init>(HiveSchemaTool.java:96)
at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1473)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Caused by: com.ctc.wstx.exc.WstxParsingException: Unexpected close tag </property>; expected </value>.
at [row,col,system-id]: [8,11,"file:/opt/module/hive-3.1.2/conf/hive-site.xml"]
at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:621)
at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:491)
at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:475)
at com.ctc.wstx.sr.BasicStreamReader.reportWrongEndElem(BasicStreamReader.java:3365)
at com.ctc.wstx.sr.BasicStreamReader.readEndElem(BasicStreamReader.java:3292)
at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2911)
at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1123)
at org.apache.hadoop.conf.Configuration$Parser.parseNext(Configuration.java:3320)
        at org.apache.hadoop.conf.Configuration$Parser.parse(Configuration.java:3114)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3007)
... 15 more

原因配置文件错误

检查标签是否有不闭合情况

检查property的值是否错误

检查value的值是否错误

vim $HIVE_HOME/conf/hive-site.xml

报错 #2 《mysql已经初始化》

Error: Table 'CTLGS' already exists (state=42S01,code=1050)
Closing: 0: jdbc:mysql://master:3306/metastore?createDatabaseIfNotExist=true
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!
Underlying cause: java.io.IOException : Schema script failed, errorcode 2
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!
at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:594)
at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:567)
at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1517)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Caused by: java.io.IOException: Schema script failed, errorcode 2
at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:1226)
at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:1204)
at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:590)
... 8 more
*** schemaTool failed ***

当mysql已经有metastore数据后想要重新初始化需要删除metastore再初始化

解决办法

mysql -u root -p

drop database metastore;

exit

bin/schematool -dbType mysql -initSchema -verbose
# 成功!

报错 #3 《hiveserver2 启动报错》

2023-11-14 14:50:06,105 WARN [main] server.HiveServer2 (HiveServer2.java:startHiveServer2(1064)) - Error starting HiveServer2 on attempt 1, will retry in 60000ms
java.lang.NoClassDefFoundError: org/apache/tez/dag/api/TezConfiguration
at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession$AbstractTriggerValidator.startTriggerValidator(TezSessionPoolSession.java:74)
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.initTriggers(TezSessionPoolManager.java:207)
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.startPool(TezSessionPoolManager.java:114)
        at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:839)
        at org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:822)
        at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:745)
        at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1037)
        at org.apache.hive.service.server.HiveServer2.access$1600(HiveServer2.java:140)
        at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1305)
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1149)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Caused by: java.lang.ClassNotFoundException: org.apache.tez.dag.api.TezConfiguration
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 16 more

原因端口10000被占用结束占用的端口或者直接重启

标签：Hive Hadoop MySQL 大数据部署

返回所有文章下一篇：HBase 集群部署指南

加载评论中...

Hive 完整部署指南

1 配置MySQL元数据

HiveServer2 部署

metastore 独立服务模式部署

部署完成！

报错合集

报错 #1 《配置文件错误》

报错 #2 《mysql已经初始化》

报错 #3 《hiveserver2 启动报错》

评论

发表评论