当前位置:首页 > 开发 > 开源软件 > 正文

【Hadoop十七】HDFS HA配置

发表于: 2015-06-13   作者:bit1129   来源:转载   浏览:
摘要: 基于Zookeeper的HDFS HA配置主要涉及两个文件,core-site和hdfs-site.xml。   测试环境有三台 hadoop.master hadoop.slave1 hadoop.slave2   hadoop.master包含的组件NameNode, JournalNode, Zookeeper,DFSZKFailoverController

基于Zookeeper的HDFS HA配置主要涉及两个文件,core-site和hdfs-site.xml。

 

测试环境有三台

hadoop.master

hadoop.slave1

hadoop.slave2

 

hadoop.master包含的组件NameNode, JournalNode, Zookeeper,DFSZKFailoverController

hadoop.slave1 包含的组件Standby NameNode, DataNode, JournaleNode,DFSZKFailoverController

hadoop.slave2 包含的组件DataNode,JournalNode

 

1. core-site.xml配置

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hdfsHA</value>
    </property>
    <property>
        <name>io.file.buffer.size</name>
        <value>131702</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:/home/hadoop/data/tmp</value>
    </property>
    <property>
        <name>ha.zookeeper.quorum</name>
        <value>hadoop.master:2181</value>
    </property>
    <property>
        <name>hadoop.proxyuser.hadoop.hosts</name>
        <value></value>
    </property>
    <property>
        <name>hadoop.proxyuser.hadoop.groups</name>
        <value></value>
    </property>
    <property>
        <name>hadoop.native.lib</name>
        <value>true</value>
        <description>Should native hadoop libraries, if present, be used.</description>
    </property>
</configuration>

 

 

 

2. hdfs-site.xml配置

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->
<configuration>
    <property>
	<name>dfs.nameservices</name>
        <value>hdfsHA</value>
    </property>
    <property>
        <name>dfs.ha.namenodes.hdfsHA</name>
        <value>nn1,nn2</value>
    </property>
    <property>
	<name>dfs.namenode.rpc-address.hdfsHA.nn1</name>
	<value>hadoop.master:9000</value>
    </property>
    <property>
	<name>dfs.namenode.rpc-address.hdfsHA.nn2</name>
	<value>hadoop.slave1:9000</value>
    </property>

    <property>
	<name>dfs.namenode.http-address.hdfsHA.nn1</name>
	<value>hadoop.master:50070</value>
    </property>
    <property>
        <name>dfs.namenode.http-address.hdfsHA.nn2</name>
        <value>hadoop.slave1:50070</value>
    </property>
    <property>
	<name>dfs.namenode.shared.edits.dir</name>
	<value>qjournal://hadoop.master:8485;hadoop.slave1:8485;hadoop.slave2:8485/hdfsHA</value>
    </property>
    <property>
	<name>dfs.ha.automatic-failover.enabled.hdfsHA</name>
	<value>true</value>
    </property>
    <property>
        <name>dfs.client.failover.proxy.provider.hdfsHA</name>  
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>  
    </property>  
  
    <property>  
        <name>dfs.journalnode.edits.dir</name>  
        <value>/home/hadoop/data/dfs/journal</value>  
    </property>  
  
    <property>  
        <name>dfs.ha.fencing.methods</name>  
        <value>sshfence</value>  
    </property>  
  
    <property>  
        <name>dfs.ha.fencing.ssh.private-key-files</name>  
        <value>/home/hadoop/.ssh/id_rsa</value>  
    </property>  



    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/home/hadoop/data/dfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/home/hadoop/data/dfs/data</value>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>2</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>hadoop.master:9001</value>
    </property>
    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
    </property>
</configuration>

 

 

 

3.启动过程

3.1 将两个配置文件分发到hadoop.slave1和hadoop.slave2节点

3.2 在三台机器上启动journalnode

 

 

sbin/hadoop-daemon.sh start journalnode
 

 

启动进程为6725 org.apache.hadoop.hdfs.qjournal.server.JournalNode

 

3.3 在hadoop.master上格式化Zookeeper(实际上三台机器哪一台都可以)

 

 

bin/hdfs zkfc  -formatZK
成功信息为:ha.ActiveStandbyElector: Successfully created /hadoop-ha/hdfsHA in ZK

 

3.4  在hadoop.master上初始化namenode并启动

 

 

bin/hdfs namenode  -format
sbin/hadoop-daemon.sh  start namenode

 

3.5 对hadoop.slave1 namenode进行格式化并启动

 

 

bin/hdfs namenode  -format
sbin/hadoop-daemon.sh  start namenode
 

 

此时,两台机器都处于standby状态

 

3.6 在hadoop.master和hadoop.slave1上启动zkfc

 

sbin/hadoop-daemon.sh   start  zkfc
 

 

启动进程为DFSZKFailoverController

 

 

此时,有一台处于active状态,另一台处于standby状态

 

3.7 在hadoop.master上启动datanode,此时slave1和slave2两台机器的datanode启动

【Hadoop十七】HDFS HA配置

  • 0

    开心

    开心

  • 0

    板砖

    板砖

  • 0

    感动

    感动

  • 0

    有用

    有用

  • 0

    疑问

    疑问

  • 0

    难过

    难过

  • 0

    无聊

    无聊

  • 0

    震惊

    震惊

编辑推荐
前提条件 先搭建 http://www.cnblogs.com/raphael5200/p/5152004.html 的环境,然后在其基础上进行
Hadoop中的NameNode好比是人的心脏,非常重要,绝对不可以停止工作。在hadoop1时代,只有一个NameNo
Hadoop 2.0 产生的背景 Hadoop 1.0 中HDFS和MapReduce存在高可用和扩展方面的问题   HDFS存在的问
一、服务器分布及相关说明 1、服务器角色 2、Hadoop(HDFS HA)总体架构 <p style="color: #2c2c
1.HA的简介 Background Prior to Hadoop 2.0.0, the NameNode was a single point of failure (SPOF
部署逻辑架构: HDFS HA部署物理架构 <div style="font-family:'Mi
1. HDFS 2.0 基本概念 相比于 Hadoop 1.0,Hadoop 2.0 中的 HDFS 增加了两个重大特性,HA 和 Federaio
1. HDFS 2.0 基本概念 相比于 Hadoop 1.0,Hadoop 2.0 中的 HDFS 增加了两个重大特性,HA 和 Federaio
1. HDFS 2.0 基本概念 相比于 Hadoop 1.0,Hadoop 2.0 中的 HDFS 增加了两个重大特性,HA 和 Federaio
step1:将安装包hadoop-2.2.0.tar.gz存放到某一个目录下,并解压 step2:修改解压后的目录中的文件夹/
版权所有 IT知识库 CopyRight © 2009-2015 IT知识库 IT610.com , All Rights Reserved. 京ICP备09083238号