博客
关于我
强烈建议你试试无所不能的chatGPT,快点击我
Linux下Hive安装
阅读量:5828 次
发布时间:2019-06-18

本文共 6878 字,大约阅读时间需要 22 分钟。

hot3.png

1、下载

2、安装Hive之前,请确保、MySQL和已安装完毕

3、解压apache-hive-2.1.1-bin.tar.gz,并拷贝到/usr/local目录

tar -zxvf apache-hive-2.1.1-bin.tar.gzmv apache-hive-2.1.1-bin /app/hive

4、配置Hive环境变量,vi /etc/profile,追加以下内容,保存后执行source /etc/profile

HIVE_HOME=/app/hiveexport PATH=$PATH:$HIVE_HOME/bin

5、修改Hive配置文件,conf/目录下

cp hive-env.sh.template hive-env.shcp hive-default.xml.template hive-site.xmlcp hive-log4j2.properties.template hive-log4j2.propertiescp hive-exec-log4j2.properties.template hive-exec-log4j2.propertiescp llap-cli-log4j2.properties.template llap-cli-log4j2.propertiescp llap-daemon-log4j2.properties.template llap-daemon-log4j2.properties

    1. 修改hive-env.sh

export JAVA_HOME=/app/jdk1.8.0_111export HADOOP_HOME=/app/hadoopexport HIVE_HOME=/app/hiveexport HIVE_CONF_DIR=$HIVE_HOME/conf

    2. 修改hive-site.xml,修改数据库连接参数,这里采用MySQL

javax.jdo.option.ConnectionURL
jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true&characterEncoding=UTF-8&useSSL=false
javax.jdo.option.ConnectionDriverName
com.mysql.jdbc.Driver
javax.jdo.option.ConnectionUserName
root
javax.jdo.option.ConnectionPassword
123456

6、将mysql驱动mysql-connector-java-5.1.34.jar拷贝到${HIVE_HOME}/lib目录下

7、初始化元素数据,这里为mysql,默认derby

./schematool -initSchema -dbType mysql

8、启动hive

Hive

 

 

 

常见问题:

    Q:Hive报错信息如下:

         java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7B

    A:修改hive-site.xml,添加${system:java.io.tmpdir}和${system:user.name}

system:java.io.tmpdir
/usr/local/hive-2.1.1/tmp
system:user.name
hive

    Q:Hive报错信息如下:

          Caused by: MetaException(message:Version information not found in metastore. )

    A:修改hive-site.xml,将hive.metastore.schema.verification属性值为false,该参数防止架构版本不兼容时的 Metastore 操作。考虑将此设置为“True”,以减少 Metastore 操作期间发生架构损坏的可能性防止架构版本不兼容时的 Metastore 操作。考虑将此设置为“True”,以减少 Metastore 操作期间发生架构损坏的可能性。

    Q:Hive报错信息如下:

         Caused by: java.lang.RuntimeException: Unable to instantiate 

                         org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

    A:打开MySQL,执行如下语句:

use mysql;set global binlog_format='MIXED';

   Q:Hive报错信息如下:

         Caused by: MetaException(message:Hive metastore database is not initialized. Please use schematool (e.g. ./schematool -initSchema -dbType ...) to create the schema. If needed, don't forget to include the option to auto-create the underlying database in your JDBC connection string (e.g. ?createDatabaseIfNotExist=true for mysql))

    A:执行schema初始化(以MySQL为例)

./schematool -initSchema -dbType mysql

   Q:Hive报错信息如下:

         Container [pid=3680,containerID=container_1483878515419_0001_01_000005] is running beyond virtual memory limits. Current usage: 75.0 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.

    A:修改yarn-site.xml文件中的yarn.nodemanager.vmem-pmem-ratio属性值(默认为2:1,物理内存与虚拟内存比例)

yarn.nodemanager.vmem-pmem-ratio
10

        具体错误信息如下:

hive> create table t_loginfo as select * from loginfo;WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.Query ID = root_20170108203354_8b2827b5-0d4b-4d5b-8fff-ceaa784e9030Total jobs = 3Launching Job 1 out of 3Number of reduce tasks is set to 0 since there's no reduce operatorStarting Job = job_1483878515419_0001, Tracking URL = http://localhost:8088/proxy/application_1483878515419_0001/Kill Command = /usr/local/install_path/hadoop-2.7.3/bin/hadoop job  -kill job_1483878515419_0001Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 02017-01-08 20:35:23,745 Stage-1 map = 0%,  reduce = 0%2017-01-08 20:35:46,498 Stage-1 map = 100%,  reduce = 0%2017-01-08 20:35:47,733 Stage-1 map = 0%,  reduce = 0%2017-01-08 20:35:57,498 Stage-1 map = 100%,  reduce = 0%Ended Job = job_1483878515419_0001 with errorsError during job, obtaining debugging information...Examining task ID: task_1483878515419_0001_m_000000 (and more) from job job_1483878515419_0001Task with the most failures(4): -----Task ID:  task_1483878515419_0001_m_000000URL:  http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1483878515419_0001&tipid=task_1483878515419_0001_m_000000-----Diagnostic Messages for this Task:Container [pid=3680,containerID=container_1483878515419_0001_01_000005] is running beyond virtual memory limits. Current usage: 75.0 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.Dump of the process-tree for container_1483878515419_0001_01_000005 :	|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE	|- 3680 3678 3680 3680 (bash) 1 0 108679168 351 /bin/bash -c /usr/local/install_path/jdk1.8.0_111/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Xmx200m -Djava.io.tmpdir=/usr/local/install_path/hadoop-2.7.3/tmp/nm-local-dir/usercache/root/appcache/application_1483878515419_0001/container_1483878515419_0001_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/install_path/hadoop-2.7.3/logs/userlogs/application_1483878515419_0001/container_1483878515419_0001_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 57321 attempt_1483878515419_0001_m_000000_3 5 1>/usr/local/install_path/hadoop-2.7.3/logs/userlogs/application_1483878515419_0001/container_1483878515419_0001_01_000005/stdout 2>/usr/local/install_path/hadoop-2.7.3/logs/userlogs/application_1483878515419_0001/container_1483878515419_0001_01_000005/stderr  	|- 3685 3680 3680 3680 (java) 370 25 2258333696 18847 /usr/local/install_path/jdk1.8.0_111/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx200m -Djava.io.tmpdir=/usr/local/install_path/hadoop-2.7.3/tmp/nm-local-dir/usercache/root/appcache/application_1483878515419_0001/container_1483878515419_0001_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/install_path/hadoop-2.7.3/logs/userlogs/application_1483878515419_0001/container_1483878515419_0001_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 57321 attempt_1483878515419_0001_m_000000_3 5 Container killed on request. Exit code is 143Container exited with a non-zero exit code 143FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTaskMapReduce Jobs Launched: Stage-Stage-1: Map: 1   HDFS Read: 0 HDFS Write: 0 FAILTotal MapReduce CPU Time Spent: 0 msec

 

转载于:https://my.oschina.net/u/3209870/blog/820769

你可能感兴趣的文章
Atitit 记录方法调用参数上下文arguments
查看>>
webstorm常用功能FTP,及常用快捷键
查看>>
eclipse html 打开方式
查看>>
[求助] win7 x64 封装 出现 Administrator.xxxxx 的问题
查看>>
人类投资经理再也无法击败电脑的时代终将到来了...
查看>>
一个最小手势库的实现
查看>>
HoloLens开发手记 - Vuforia开发概述 Vuforia development overview
查看>>
Android支付之支付宝封装类
查看>>
<亲测>CentOS中yum安装ffmpeg
查看>>
【分享】马化腾:产品设计与用户体验
查看>>
【机器学习PAI实践十】深度学习Caffe框架实现图像分类的模型训练
查看>>
全智慧的网络:思科十年来最具颠覆性的创新
查看>>
怎样将现有应用迁移到 VMware NSX
查看>>
赛门铁克收购以色列移动安全初创公司Skycure 旨在构建网络安全防御平台
查看>>
《Photoshop蒙版与合成(第2版)》目录—导读
查看>>
“最佳人气奖”出炉!4月27号,谁能拿到阿里聚安全算法挑战赛的桂冠?
查看>>
《网页美工设计Photoshop+Flash+Dreamweaver从入门到精通》——2.6 图层与图层样式...
查看>>
《iOS组件与框架——iOS SDK高级特性剖析》——第2章,第2.7节获取线路
查看>>
Spring中 @Autowired标签与 @Resource标签 的区别
查看>>
人工智能凭什么毁灭人类
查看>>