hadoop集群搭建之HA模式

1.Hadoop环境变量相关配置

1.在node01上创建hadoop安装路径：
mkdir /opt/bigdata

2.解压hadoop软件包
   tar xf hadoop-2.6.5.tar.gz
   mv hadoop-2.6.5 /opt/bigdata/
3.配置hadoop环境便令
   vi /etc/profile
     export JAVA_HOME=/usr/java/default
     export HADOOP_HOME=/opt/bigdata/hadoop-2.6.5
     export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

4.使配置生效
source /etc/profile

5.检测

在任意目录输入hd按Tab建看是否能联想出hdfs命令，有侧安装配置成功

6.从node01分发到其他节点（node02，node03，node04）

        cd /opt
           scp -r ./bigdata/ node02:`pwd`
           scp -r ./bigdata/ node03:`pwd`
           scp -r ./bigdata/ node04:`pwd`

2.安装zookeeper（node02，node03，node04）

node02:
           tar xf zook....tar.gz
           mv zoo... /opt/bigdata
           cd /opt/bigdata/zoo....
           cd conf
           cp zoo_sample.cfg zoo.cfg
           vi zoo.cfg
               datadir=/var/bigdata/hadoop/zk
               server.1=node02:2888:3888
               server.2=node03:2888:3888
               server.3=node04:2888:3888
           mkdir /var/bigdata/hadoop/zk
           echo 1 > /var/bigdata/hadoop/zk/myid
           vi /etc/profile
               export ZOOKEEPER_HOME=/opt/bigdata/zookeeper-3.4.6
               export                 PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin
           . /etc/profile
           cd /opt/bigdata
           scp -r ./zookeeper-3.4.6 node03:`pwd`
           scp -r ./zookeeper-3.4.6 node04:`pwd`
       node03:
           mkdir /var/bigdata/hadoop/zk
           echo 2 > /var/bigdata/hadoop/zk/myid
           *环境变量
           . /etc/profile
       node04:
           mkdir /var/bigdata/hadoop/zk
           echo 3 > /var/bigdata/hadoop/zk/myid
           *环境变量
           . /etc/profile

node02~node04:
zkServer.sh start

3.hadoop文件相关配置

1.修改hadoop-env.sh

vi hadoop-env.sh
export JAVA_HOME=/usr/java/default

2.修改core-site.xml

vi core-site.xml

<property><name>fs.defaultFS</name><value>hdfs://mycluster</value>
</property><property><name>ha.zookeeper.quorum</name><value>node02:2181,node03:2181,node04:2181</value>
</property>

3.修改hdfs-site.xml

vi hdfs-site.xml

<property><name>dfs.nameservices</name><value>mycluster</value></property><property><name>dfs.ha.namenodes.mycluster</name><value>nn1,nn2</value></property><property><name>dfs.namenode.rpc-address.mycluster.nn1</name><value>node01:8020</value></property><property><name>dfs.namenode.rpc-address.mycluster.nn2</name><value>node02:8020</value></property><property><name>dfs.namenode.http-address.mycluster.nn1</name><value>node01:50070</value></property><property><name>dfs.namenode.http-address.mycluster.nn2</name><value>node02:50070</value></property>#以下是JN在哪里启动，数据存那个磁盘<property><name>dfs.namenode.shared.edits.dir</name><value>qjournal://node01:8485;node02:8485;node03:8485/mycluster</value></property><property><name>dfs.journalnode.edits.dir</name><value>/var/bigdata/hadoop/ha/dfs/jn</value></property>#HA角色切换的代理类和实现方法，我们用的ssh免密<property><name>dfs.client.failover.proxy.provider.mycluster</name><value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value></property><property><name>dfs.ha.fencing.methods</name><value>sshfence</value></property><property><name>dfs.ha.fencing.ssh.private-key-files</name><value>/root/.ssh/id_dsa</value></property>#开启自动化： 启动zkfc<property><name>dfs.ha.automatic-failover.enabled</name><value>true</value></property>

4.修改vi slaves

vi slaves