1、下载
2、安装Hive之前,请确保、MySQL和已安装完毕
3、解压apache-hive-2.1.1-bin.tar.gz,并拷贝到/usr/local目录
tar -zxvf apache-hive-2.1.1-bin.tar.gzmv apache-hive-2.1.1-bin /app/hive
4、配置Hive环境变量,vi /etc/profile,追加以下内容,保存后执行source /etc/profile
HIVE_HOME=/app/hiveexport PATH=$PATH:$HIVE_HOME/bin
5、修改Hive配置文件,conf/目录下
cp hive-env.sh.template hive-env.shcp hive-default.xml.template hive-site.xmlcp hive-log4j2.properties.template hive-log4j2.propertiescp hive-exec-log4j2.properties.template hive-exec-log4j2.propertiescp llap-cli-log4j2.properties.template llap-cli-log4j2.propertiescp llap-daemon-log4j2.properties.template llap-daemon-log4j2.properties
1. 修改hive-env.sh
export JAVA_HOME=/app/jdk1.8.0_111export HADOOP_HOME=/app/hadoopexport HIVE_HOME=/app/hiveexport HIVE_CONF_DIR=$HIVE_HOME/conf
2. 修改hive-site.xml,修改数据库连接参数,这里采用MySQL
javax.jdo.option.ConnectionURL jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true&characterEncoding=UTF-8&useSSL=false javax.jdo.option.ConnectionDriverName com.mysql.jdbc.Driver javax.jdo.option.ConnectionUserName root javax.jdo.option.ConnectionPassword 123456
6、将mysql驱动mysql-connector-java-5.1.34.jar拷贝到${HIVE_HOME}/lib目录下
7、初始化元素数据,这里为mysql,默认derby
./schematool -initSchema -dbType mysql
8、启动hive
Hive
常见问题:
Q:Hive报错信息如下:
java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7B
A:修改hive-site.xml,添加${system:java.io.tmpdir}和${system:user.name}
system:java.io.tmpdir /usr/local/hive-2.1.1/tmp system:user.name hive
Q:Hive报错信息如下:
Caused by: MetaException(message:Version information not found in metastore. )
A:修改hive-site.xml,将hive.metastore.schema.verification属性值为false,该参数防止架构版本不兼容时的 Metastore 操作。考虑将此设置为“True”,以减少 Metastore 操作期间发生架构损坏的可能性防止架构版本不兼容时的 Metastore 操作。考虑将此设置为“True”,以减少 Metastore 操作期间发生架构损坏的可能性。
Q:Hive报错信息如下:
Caused by: java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
A:打开MySQL,执行如下语句:
use mysql;set global binlog_format='MIXED';
Q:Hive报错信息如下:
Caused by: MetaException(message:Hive metastore database is not initialized. Please use schematool (e.g. ./schematool -initSchema -dbType ...) to create the schema. If needed, don't forget to include the option to auto-create the underlying database in your JDBC connection string (e.g. ?createDatabaseIfNotExist=true for mysql))
A:执行schema初始化(以MySQL为例)
./schematool -initSchema -dbType mysql
Q:Hive报错信息如下:
Container [pid=3680,containerID=container_1483878515419_0001_01_000005] is running beyond virtual memory limits. Current usage: 75.0 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.
A:修改yarn-site.xml文件中的yarn.nodemanager.vmem-pmem-ratio属性值(默认为2:1,物理内存与虚拟内存比例)
yarn.nodemanager.vmem-pmem-ratio 10
具体错误信息如下:
hive> create table t_loginfo as select * from loginfo;WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.Query ID = root_20170108203354_8b2827b5-0d4b-4d5b-8fff-ceaa784e9030Total jobs = 3Launching Job 1 out of 3Number of reduce tasks is set to 0 since there's no reduce operatorStarting Job = job_1483878515419_0001, Tracking URL = http://localhost:8088/proxy/application_1483878515419_0001/Kill Command = /usr/local/install_path/hadoop-2.7.3/bin/hadoop job -kill job_1483878515419_0001Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 02017-01-08 20:35:23,745 Stage-1 map = 0%, reduce = 0%2017-01-08 20:35:46,498 Stage-1 map = 100%, reduce = 0%2017-01-08 20:35:47,733 Stage-1 map = 0%, reduce = 0%2017-01-08 20:35:57,498 Stage-1 map = 100%, reduce = 0%Ended Job = job_1483878515419_0001 with errorsError during job, obtaining debugging information...Examining task ID: task_1483878515419_0001_m_000000 (and more) from job job_1483878515419_0001Task with the most failures(4): -----Task ID: task_1483878515419_0001_m_000000URL: http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1483878515419_0001&tipid=task_1483878515419_0001_m_000000-----Diagnostic Messages for this Task:Container [pid=3680,containerID=container_1483878515419_0001_01_000005] is running beyond virtual memory limits. Current usage: 75.0 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.Dump of the process-tree for container_1483878515419_0001_01_000005 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 3680 3678 3680 3680 (bash) 1 0 108679168 351 /bin/bash -c /usr/local/install_path/jdk1.8.0_111/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx200m -Djava.io.tmpdir=/usr/local/install_path/hadoop-2.7.3/tmp/nm-local-dir/usercache/root/appcache/application_1483878515419_0001/container_1483878515419_0001_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/install_path/hadoop-2.7.3/logs/userlogs/application_1483878515419_0001/container_1483878515419_0001_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 57321 attempt_1483878515419_0001_m_000000_3 5 1>/usr/local/install_path/hadoop-2.7.3/logs/userlogs/application_1483878515419_0001/container_1483878515419_0001_01_000005/stdout 2>/usr/local/install_path/hadoop-2.7.3/logs/userlogs/application_1483878515419_0001/container_1483878515419_0001_01_000005/stderr |- 3685 3680 3680 3680 (java) 370 25 2258333696 18847 /usr/local/install_path/jdk1.8.0_111/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx200m -Djava.io.tmpdir=/usr/local/install_path/hadoop-2.7.3/tmp/nm-local-dir/usercache/root/appcache/application_1483878515419_0001/container_1483878515419_0001_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/install_path/hadoop-2.7.3/logs/userlogs/application_1483878515419_0001/container_1483878515419_0001_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 57321 attempt_1483878515419_0001_m_000000_3 5 Container killed on request. Exit code is 143Container exited with a non-zero exit code 143FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTaskMapReduce Jobs Launched: Stage-Stage-1: Map: 1 HDFS Read: 0 HDFS Write: 0 FAILTotal MapReduce CPU Time Spent: 0 msec