2023年3月10日
Hive 最小化部署指南
大数据 · Hive
详细介绍 Hive 最小化部署的步骤,包括上传解压、环境变量配置、初始化元数据库等操作,适用于学习和测试环境
最小化
无需启动 Hive服务 元数据库
需要启动 HDFS
用于学习和测试环境 不使用生产环境
只需要安装在master机器
上传解压
# 改成你自己的路径和IP 照抄别逼我骂你 scp apache-hive-3.1.2-bin.tar.gz root@192.168.25.10:/opt/softs/ cd /opt/softs/ # 解压 tar -xvf apache-hive-3.1.2-bin.tar.gz # 移动到 module mv apache-hive-3.1.2-bin ../module/ cd ../module/ # 重命名 mv apache-hive-3.1.2-bin/ hive-3.1.2
# 改成你自己的路径和IP 照抄别逼我骂你 scp apache-hive-3.1.2-bin.tar.gz root@192.168.25.10:/opt/softs/ cd /opt/softs/ # 解压 tar -xvf apache-hive-3.1.2-bin.tar.gz # 移动到 module mv apache-hive-3.1.2-bin ../module/ cd ../module/ # 重命名 mv apache-hive-3.1.2-bin/ hive-3.1.2
# 改成你自己的路径和IP 照抄别逼我骂你 scp apache-hive-3.1.2-bin.tar.gz root@192.168.25.10:/opt/softs/ cd /opt/softs/ # 解压 tar -xvf apache-hive-3.1.2-bin.tar.gz # 移动到 module mv apache-hive-3.1.2-bin ../module/ cd ../module/ # 重命名 mv apache-hive-3.1.2-bin/ hive-3.1.2
添加环境变量
vim /etc/profile # 在最下面添加 # hive export HIVE_HOME=/opt/module/hive-3.1.2 export PATH=$PATH:$HIVE_HOME/bin :wq # 保存退出 source /etc/profile # 刷新环境变量 cd $HIVE_HOME # 检查是否成功
vim /etc/profile # 在最下面添加 # hive export HIVE_HOME=/opt/module/hive-3.1.2 export PATH=$PATH:$HIVE_HOME/bin :wq # 保存退出 source /etc/profile # 刷新环境变量 cd $HIVE_HOME # 检查是否成功
vim /etc/profile # 在最下面添加 # hive export HIVE_HOME=/opt/module/hive-3.1.2 export PATH=$PATH:$HIVE_HOME/bin :wq # 保存退出 source /etc/profile # 刷新环境变量 cd $HIVE_HOME # 检查是否成功
初始化源数据库
bin/schematool -dbType derby -initSchema
bin/schematool -dbType derby -initSchema
bin/schematool -dbType derby -initSchema
初始化报错看最下面部分
报错 #1 《guava冲突》有解决办法
启动
./bin/hive
./bin/hive
./bin/hive
测试创建表和插入数据
create table student(id int, name string); insert into student values (1, "dbp");
create table student(id int, name string); insert into student values (1, "dbp");
create table student(id int, name string); insert into student values (1, "dbp");
查看HDFS
表会被映射到 HDFS中
报错合集
使用 有右上角🔍搜索功能 搜索你遇到的报错
报错 #1 《guava冲突》
[root@master hive-3.1.2]# bin/schematool -dbType derby -initSchema SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/module/hadoop-3.1.3/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/module/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2023-11-13 16:13:33,731 INFO [main] conf.HiveConf (HiveConf.java:findConfigFile(187)) - Found configuration file null Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357) at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338) at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:518) at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:536) at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:430) at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5141) at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5104) at org.apache.hive.beeline.HiveSchemaTool.<init>(HiveSchemaTool.java:96) at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1473) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:318) at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
[root@master hive-3.1.2]# bin/schematool -dbType derby -initSchema SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/module/hadoop-3.1.3/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/module/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2023-11-13 16:13:33,731 INFO [main] conf.HiveConf (HiveConf.java:findConfigFile(187)) - Found configuration file null Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357) at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338) at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:518) at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:536) at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:430) at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5141) at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5104) at org.apache.hive.beeline.HiveSchemaTool.<init>(HiveSchemaTool.java:96) at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1473) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:318) at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
[root@master hive-3.1.2]# bin/schematool -dbType derby -initSchema SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/module/hadoop-3.1.3/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/module/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2023-11-13 16:13:33,731 INFO [main] conf.HiveConf (HiveConf.java:findConfigFile(187)) - Found configuration file null Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357) at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338) at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:518) at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:536) at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:430) at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5141) at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5104) at org.apache.hive.beeline.HiveSchemaTool.<init>(HiveSchemaTool.java:96) at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1473) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:318) at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
找重点
搜索问题找到文章:
https://blog.csdn.net/qq_44766883/article/details/108582781
解决办法
删除guava.jar
[root@master hive-3.1.2]# ls lib/guava* lib/guava-19.0.jar # 删除搜索到的guava包 [root@master hive-3.1.2]# rm lib/guava-19.0.jar rm:是否删除普通文件 "lib/guava-19.0.jar"?y # 复制hadoop guava包 到 hive/lib cp $HADOOP_HOME/share/hadoop/hdfs/lib/guava-27.0-jre.jar ./lib
[root@master hive-3.1.2]# ls lib/guava* lib/guava-19.0.jar # 删除搜索到的guava包 [root@master hive-3.1.2]# rm lib/guava-19.0.jar rm:是否删除普通文件 "lib/guava-19.0.jar"?y # 复制hadoop guava包 到 hive/lib cp $HADOOP_HOME/share/hadoop/hdfs/lib/guava-27.0-jre.jar ./lib
[root@master hive-3.1.2]# ls lib/guava* lib/guava-19.0.jar # 删除搜索到的guava包 [root@master hive-3.1.2]# rm lib/guava-19.0.jar rm:是否删除普通文件 "lib/guava-19.0.jar"?y # 复制hadoop guava包 到 hive/lib cp $HADOOP_HOME/share/hadoop/hdfs/lib/guava-27.0-jre.jar ./lib
原因
guava版本冲突
报错 #2 《初始化失败》
[root@master hive-3.1.2]# bin/schematool -dbType derby -initSchema SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/module/hadoop-3.1.3/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/module/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2023-11-13 20:43:30,604 INFO [main] conf.HiveConf (HiveConf.java:findConfigFile(187)) - Found configuration file null 2023-11-13 20:43:30,772 INFO [main] tools.HiveSchemaHelper (HiveSchemaHelper.java:logAndPrintToStdout(117)) - Metastore connection URL: jdbc:derby:;databaseName=metastore_db;create=true Metastore connection URL: jdbc:derby:;databaseName=metastore_db;create=true 2023-11-13 20:43:30,773 INFO [main] tools.HiveSchemaHelper (HiveSchemaHelper.java:logAndPrintToStdout(117)) - Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver 2023-11-13 20:43:30,773 INFO [main] tools.HiveSchemaHelper (HiveSchemaHelper.java:logAndPrintToStdout(117)) - Metastore connection User: APP Metastore connection User: APP Starting metastore schema initialization to 3.1.0 Initialization script hive-schema-3.1.0.derby.sql Error: FUNCTION 'NUCLEUS*ASCII' already exists. (state=X0Y68,code=30000) org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !! Underlying cause: java.io.IOException : Schema script failed, errorcode 2 Use --verbose for detailed stacktrace. *** schemaTool failed
[root@master hive-3.1.2]# bin/schematool -dbType derby -initSchema SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/module/hadoop-3.1.3/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/module/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2023-11-13 20:43:30,604 INFO [main] conf.HiveConf (HiveConf.java:findConfigFile(187)) - Found configuration file null 2023-11-13 20:43:30,772 INFO [main] tools.HiveSchemaHelper (HiveSchemaHelper.java:logAndPrintToStdout(117)) - Metastore connection URL: jdbc:derby:;databaseName=metastore_db;create=true Metastore connection URL: jdbc:derby:;databaseName=metastore_db;create=true 2023-11-13 20:43:30,773 INFO [main] tools.HiveSchemaHelper (HiveSchemaHelper.java:logAndPrintToStdout(117)) - Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver 2023-11-13 20:43:30,773 INFO [main] tools.HiveSchemaHelper (HiveSchemaHelper.java:logAndPrintToStdout(117)) - Metastore connection User: APP Metastore connection User: APP Starting metastore schema initialization to 3.1.0 Initialization script hive-schema-3.1.0.derby.sql Error: FUNCTION 'NUCLEUS*ASCII' already exists. (state=X0Y68,code=30000) org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !! Underlying cause: java.io.IOException : Schema script failed, errorcode 2 Use --verbose for detailed stacktrace. *** schemaTool failed
[root@master hive-3.1.2]# bin/schematool -dbType derby -initSchema SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/module/hadoop-3.1.3/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/module/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2023-11-13 20:43:30,604 INFO [main] conf.HiveConf (HiveConf.java:findConfigFile(187)) - Found configuration file null 2023-11-13 20:43:30,772 INFO [main] tools.HiveSchemaHelper (HiveSchemaHelper.java:logAndPrintToStdout(117)) - Metastore connection URL: jdbc:derby:;databaseName=metastore_db;create=true Metastore connection URL: jdbc:derby:;databaseName=metastore_db;create=true 2023-11-13 20:43:30,773 INFO [main] tools.HiveSchemaHelper (HiveSchemaHelper.java:logAndPrintToStdout(117)) - Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver 2023-11-13 20:43:30,773 INFO [main] tools.HiveSchemaHelper (HiveSchemaHelper.java:logAndPrintToStdout(117)) - Metastore connection User: APP Metastore connection User: APP Starting metastore schema initialization to 3.1.0 Initialization script hive-schema-3.1.0.derby.sql Error: FUNCTION 'NUCLEUS*ASCII' already exists. (state=X0Y68,code=30000) org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !! Underlying cause: java.io.IOException : Schema script failed, errorcode 2 Use --verbose for detailed stacktrace. *** schemaTool failed
解决办法
当初始化过 想重新初始化 需要删掉 metastore_db 才能再次初始化
rm -rf metastore_db
rm -rf metastore_db
rm -rf metastore_db