Skip to content

Latest commit

 

History

History
149 lines (103 loc) · 5.81 KB

Hive集群搭建及操作.md

File metadata and controls

149 lines (103 loc) · 5.81 KB

Hive集群搭建及操作

  • 在Hive中:表对应HDFS的目录(或文件夹);表中的数据对应HDFS的文件

一、上传

二、安装Hive(Hive只在一个节点上安装即可)

  • 解压apache-hive-0.13.0-bin.tar.gz

    tar -zxvf apache-hive-0.13.0-bin.tar.gz -C /softWare/
    
  • 修改配置文件

    • 将hive添加到环境变量中

      cd /etc/
      export HIVE_HOME=/softWare/apache-hive-0.13.0-bin
      export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$HBASE_HOME/bin:$HIVE_HOME/bin
      
    • 修改hive-default.xml.template为hive-site.xml

      cd /softWare/apache-hive-0.13.0-bin/conf
      mv hive-default.xml.template hive-site.xml
      
    • 修改hive-site.xml文件

      <property>
         <name>javax.jdo.option.ConnectionURL</name>
         <value>jdbc:mysql://hadoop05:3306/hive?createDatabaseIfNotExist=true</value>
         <description>JDBC connect string for a JDBC metastore</description>
       </property>
      
       <property>
         <name>javax.jdo.option.ConnectionDriverName</name>
         <value>com.mysql.jdbc.Driver</value>
         <description>Driver class name for a JDBC metastore</description>
       </property>
      
       <property>
         <name>javax.jdo.option.ConnectionUserName</name>
         <value>root</value>
         <description>username to use against metastore database</description>
       </property>
      
       <property>
         <name>javax.jdo.option.ConnectionPassword</name>
         <value>root</value>
         <description>password to use against metastore database</description>
       </property>
      
    • 将数据库连接驱动拷贝到/softWare/apache-hive-0.13.0-bin/lib里

  • mysql授权

    GRANT ALL PRIVILEGES ON hive.* TO 'root'@'%' IDENTIFIED BY 'root' WITH GRANT OPTION;
    FLUSH PRIVILEGES;
    
    GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'root' WITH GRANT OPTION;
    FLUSH PRIVILEGES;
    

    -------------------------------------操作---------------------------------------------------

    [root@hadoop05 mysql]# mysql -u root -proot
      Welcome to the MySQL monitor.  Commands end with ; or \g.
      Your MySQL connection id is 9
      Server version: 5.1.73 MySQL Community Server (GPL)
    
      Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
    
      Oracle is a registered trademark of Oracle Corporation and/or its
      affiliates. Other names may be trademarks of their respective
      owners.
    
      Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
    
    mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'root' WITH GRANT OPTION;
    Query OK, 0 rows affected (0.00 sec)
    
    mysql> FLUSH PRIVILEGES;
    Query OK, 0 rows affected (0.00 sec)
    
    mysql>
    

三、启动hive

  cd /softWare/apache-hive-0.13.0-bin/bin
  ./hive

-----------------操作------------------------

  [root@hadoop03 bin]# ./hive
  18/07/29 17:36:04 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
  18/07/29 17:36:04 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
  18/07/29 17:36:04 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
  18/07/29 17:36:04 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
  18/07/29 17:36:04 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
  18/07/29 17:36:04 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
  18/07/29 17:36:04 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
  18/07/29 17:36:04 INFO Configuration.deprecation: mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed

  Logging initialized using configuration in jar:file:/softWare/apache-hive-0.13.0-bin/lib/hive-common-0.13.0.jar!/hive-log4j.properties
  hive> 

四、使用hive(表描述在hive数据库的TBLS表中,表中的字段在COLUMNS_V2表中,表的id在CDS表中,存储HDFS上的路径在SDS表中)

  • 建表(默认是内部表)

    hive> create table heros (id bigint,name string,age int,height double) row format delimited fields terminated by '\t';
    OK
    Time taken: 0.992 seconds
    
  • 查看有哪些表

    hive> show tables;
    OK
    heros
    Time taken: 0.117 seconds, Fetched: 1 row(s)
    
  • 查看建表信息

    hive> show create table heros;
    OK
    CREATE  TABLE `heros`(
      `id` bigint, 
      `name` string, 
      `age` int, 
      `height` double)
    ROW FORMAT DELIMITED 
      FIELDS TERMINATED BY '\t' 
    STORED AS INPUTFORMAT 
      'org.apache.hadoop.mapred.TextInputFormat' 
    OUTPUTFORMAT 
      'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
    LOCATION
      'hdfs://ns1/user/hive/warehouse/heros'
    TBLPROPERTIES (
      'transient_lastDdlTime'='1532911433')
    Time taken: 0.211 seconds, Fetched: 15 row(s)