当前位置:首页 > 开发 > 系统架构 > 架构 > 正文

CentOS7x86_64编译hadoop2.4

发表于: 2014-09-16   作者:cleans   来源:转载   浏览次数:
摘要: 主要涉及到工具有:hadoop-2.4.0-src.tar.gz、Ant、Maven、JDK、GCC、CMake、openssl 第一步升级系统相关编译所需的软件(升级最新版): yum install lzo-devel zlib-devel gcc autoconf automake libtool ncurses-devel openssl-devel wget http:/

主要涉及到工具有:hadoop-2.4.0-src.tar.gz、Ant、Maven、JDK、GCC、CMake、openssl

第一步升级系统相关编译所需的软件(升级最新版):

yum install lzo-devel zlib-devel gcc autoconf automake libtool ncurses-devel openssl-devel

wget http://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.4.0/hadoop-2.4.0-src.tar.gz (源代版)

tar -zxvf hadoop-2.4.0-src.tar.gz

wget http://apache.fayea.com/apache-mirror//ant/binaries/apache-ant-1.9.4-bin.tar.gz

tar -xvf apache-ant-1.9.4-bin.tar.gz

wget http://apache.fayea.com/apache-mirror/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz

tar -xvf apache-maven-3.0.5-bin.tar.gz

vi /etc/profile

export JAVA_HOME=/usr/java/jdk1.7.0_55

export JAVA_BIN=/usr/java/jdk1.7.0_55/bin

export ANT_HOME=/home/hadoop/ant

export MVN_HOME=/home/hadoop/maven

export FINDBUGS_HOME=/home/hadoop/findbugs-2.0.3

export PATH=$PATH:$JAVA_HOME/bin:$ANT_HOME/bin:$MVN_HOME/bin:$FINDBUGS_HOME/bin

生产配置文件:

source /etc/profile

验证是否配置成功

ant –version

mvn -version

findbugs –version

验证结果:

安装protobuf(以root用户登录)

wget https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz

tar zxf protobuf-2.5.0.tar.gz

cd protobuf-2.5.0

./configure

make

make install

protoc --version

安装cmake(以root用户登录)

wget http://www.cmake.org/files/v2.8/cmake-2.8.12.2-Linux-i386.tar.gz

./bootstrap

make

make install

cmake –version

(别一种方法,直接用yum install cmake)

 

编译Hadoop

mvn package -DskipTests -Pdist,native –Dtar

此时在下载maven依赖所有包及插件

慢慢等待中……

编译成功,检查nativelib 是否编译成功

[root@master hadoop-2.4.1-src]# cd hadoop-dist/target/hadoop-2.4.1/lib/native/
[root@master native]# file libhadoop.so.1.0.0
libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0xba68c7f46259525c3aae4ebd99e1faf3b6c7e7a6, not stripped

 

代表编译成功

 

或者结果如下, 也表示编译成功。

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................ SUCCESS [5.731s]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [4.215s]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [5.122s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.548s]
[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [4.271s]
[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [8.020s]
[INFO] Apache Hadoop MiniKDC ............................. SUCCESS [7.431s]
[INFO] Apache Hadoop Auth ................................ SUCCESS [7.517s]
[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [4.727s]
[INFO] Apache Hadoop Common .............................. SUCCESS [2:53.800s]
[INFO] Apache Hadoop NFS ................................. SUCCESS [16.696s]
[INFO] Apache Hadoop Common Project ...................... SUCCESS [0.042s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [6:07.368s]
[INFO] Apache Hadoop HttpFS .............................. SUCCESS [48.810s]
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [20.154s]
[INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [9.709s]
[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.049s]
[INFO] hadoop-yarn ....................................... SUCCESS [0.138s]
[INFO] hadoop-yarn-api ................................... SUCCESS [2:00.295s]
[INFO] hadoop-yarn-common ................................ SUCCESS [1:00.256s]
[INFO] hadoop-yarn-server ................................ SUCCESS [0.076s]
[INFO] hadoop-yarn-server-common ......................... SUCCESS [21.974s]
[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [28.986s]
[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [6.791s]
[INFO] hadoop-yarn-server-applicationhistoryservice ...... SUCCESS [13.558s]
[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [28.431s]
[INFO] hadoop-yarn-server-tests .......................... SUCCESS [2.644s]
[INFO] hadoop-yarn-client ................................ SUCCESS [12.729s]
[INFO] hadoop-yarn-applications .......................... SUCCESS [0.102s]
[INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [4.878s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [3.103s]
[INFO] hadoop-yarn-site .................................. SUCCESS [0.055s]
[INFO] hadoop-yarn-project ............................... SUCCESS [6.390s]
[INFO] hadoop-mapreduce-client ........................... SUCCESS [0.211s]
[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [39.919s]
[INFO] hadoop-mapreduce-client-common .................... SUCCESS [34.197s]
[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [5.716s]
[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [18.761s]
[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [17.226s]
[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [7.617s]
[INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [3.211s]
[INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [12.571s]
[INFO] hadoop-mapreduce .................................. SUCCESS [6.483s]
[INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [10.180s]
[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [15.514s]
[INFO] Apache Hadoop Archives ............................ SUCCESS [5.243s]
[INFO] Apache Hadoop Rumen ............................... SUCCESS [12.533s]
[INFO] Apache Hadoop Gridmix ............................. SUCCESS [8.247s]
[INFO] Apache Hadoop Data Join ........................... SUCCESS [6.091s]
[INFO] Apache Hadoop Extras .............................. SUCCESS [5.339s]
[INFO] Apache Hadoop Pipes ............................... SUCCESS [13.666s]
[INFO] Apache Hadoop OpenStack support ................... SUCCESS [14.356s]
[INFO] Apache Hadoop Client .............................. SUCCESS [14.354s]
[INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [0.145s]
[INFO] Apache Hadoop Scheduler Load Simulator ............ SUCCESS [39.951s]
[INFO] Apache Hadoop Tools Dist .......................... SUCCESS [8.662s]
[INFO] Apache Hadoop Tools ............................... SUCCESS [0.035s]
[INFO] Apache Hadoop Distribution ........................ SUCCESS [1:45.654s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23:27.228s
[INFO] Finished at: Tue Sep 16 23:09:45 HKT 2014
[INFO] Final Memory: 67M/179M
[INFO] ------------------------------------------------------------------------

 

 

 

错误1

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 46.796s

[INFO] Finished at: Wed Jun 04 13:28:37 CST 2014

[INFO] Final Memory: 36M/88M

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal on project hadoop-common: Could not resolve dependencies for project org.apache.hadoop:hadoop-common:jar:2.4.0: Failure to find org.apache.commons:commons-compress:jar:1.4.1 in https://repository.apache.org/content/repositories/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots.https has elapsed or updates are forced -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

[ERROR]

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR] mvn <goals> -rf :hadoop-common

解决方法:

根据上面日志提示说找不到“org.apache.commons:commons-compress:jar:1.4.1”,

直接将本地(Windows)包复制到Linux系统中,解决了。

错误2

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 2:16.693s

[INFO] Finished at: Wed Jun 04 13:56:31 CST 2014

[INFO] Final Memory: 48M/239M

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "cmake" (in directory "/home/hadoop/hadoop-2.4.0-src/hadoop-common-project/hadoop-common/target/native"): error=2, 没有那个文件或目录

[ERROR] around Ant part ...<exec dir="/home/hadoop/hadoop-2.4.0-src/hadoop-common-project/hadoop-common/target/native" executable="cmake" failonerror="true">... @ 4:133 in /home/hadoop/hadoop-2.4.0-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml

[ERROR] -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

[ERROR]

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR] mvn <goals> -rf :hadoop-common

解决方法:

是没有安装cmake导致的,再重新安装cmake;参考《5.3.1编译环境准备》

错误3

错误提示是找不到相应的文件和不能创建目录,在网上没有相关错误(根据自己经验修改目录权限为:775,让目录有创建文件或文件夹的权限,另外最好保证hadoop编译目录有2.5G至4G的空间)

chmod -Rf 775 ./ hadoop-2.4.0-src

main:

[mkdir] Created dir: /data/hadoop/hadoop-2.4.0-src/hadoop-tools/hadoop-pipes/target/test-dir

[INFO] Executed tasks

[INFO]

[INFO] --- maven-antrun-plugin:1.7:run (make) @ hadoop-pipes ---

[INFO] Executing tasks

错误3

main:

[mkdir] Created dir: /data/hadoop/hadoop-2.4.0-src/hadoop-tools/hadoop-pipes/target/native

[exec] -- The C compiler identification is GNU 4.4.7

[exec] -- The CXX compiler identification is GNU 4.4.7

[exec] -- Check for working C compiler: /usr/bin/cc

[exec] -- Check for working C compiler: /usr/bin/cc -- works

[exec] -- Detecting C compiler ABI info

[exec] -- Detecting C compiler ABI info - done

[exec] -- Check for working CXX compiler: /usr/bin/c++

[exec] -- Check for working CXX compiler: /usr/bin/c++ -- works

[exec] -- Detecting CXX compiler ABI info

[exec] -- Detecting CXX compiler ABI info - done

[exec] CMake Error at /usr/local/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:108 (message):

[exec] Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the

[exec] system variable OPENSSL_ROOT_DIR (missing: OPENSSL_LIBRARIES

[exec] OPENSSL_INCLUDE_DIR)

[exec] Call Stack (most recent call first):

[exec] /usr/local/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:315 (_FPHSA_FAILURE_MESSAGE)

[exec] /usr/local/share/cmake-2.8/Modules/FindOpenSSL.cmake:313 (find_package_handle_standard_args)

[exec] CMakeLists.txt:20 (find_package)

[exec]

[exec]

[exec] -- Configuring incomplete, errors occurred!

[exec] See also "/data/hadoop/hadoop-2.4.0-src/hadoop-tools/hadoop-pipes/target/native/CMakeFiles/CMakeOutput.log".

[INFO] ------------------------------------------------------------------------

[INFO] Reactor Summary:

[INFO]

[INFO] Apache Hadoop Main ................................ SUCCESS [13.745s]

[INFO] Apache Hadoop Project POM ......................... SUCCESS [5.538s]

[INFO] Apache Hadoop Annotations ......................... SUCCESS [7.296s]

[INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.568s]

[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [5.858s]

[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [8.541s]

[INFO] Apache Hadoop MiniKDC ............................. SUCCESS [8.337s]

[INFO] Apache Hadoop Auth ................................ SUCCESS [7.348s]

[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [4.926s]

[INFO] Apache Hadoop Common .............................. SUCCESS [2:35.956s]

[INFO] Apache Hadoop NFS ................................. SUCCESS [18.680s]

[INFO] Apache Hadoop Common Project ...................... SUCCESS [0.059s]

[INFO] Apache Hadoop HDFS ................................ SUCCESS [5:03.525s]

[INFO] Apache Hadoop HttpFS .............................. SUCCESS [38.335s]

[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [23.780s]

[INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [8.769s]

[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.159s]

[INFO] hadoop-yarn ....................................... SUCCESS [0.134s]

[INFO] hadoop-yarn-api ................................... SUCCESS [2:07.657s]

[INFO] hadoop-yarn-common ................................ SUCCESS [1:10.680s]

[INFO] hadoop-yarn-server ................................ SUCCESS [0.165s]

[INFO] hadoop-yarn-server-common ......................... SUCCESS [24.174s]

[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [27.293s]

[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [5.177s]

[INFO] hadoop-yarn-server-applicationhistoryservice ...... SUCCESS [11.399s]

[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [28.384s]

[INFO] hadoop-yarn-server-tests .......................... SUCCESS [1.346s]

[INFO] hadoop-yarn-client ................................ SUCCESS [12.937s]

[INFO] hadoop-yarn-applications .......................... SUCCESS [0.108s]

[INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [5.303s]

[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [3.212s]

[INFO] hadoop-yarn-site .................................. SUCCESS [0.050s]

[INFO] hadoop-yarn-project ............................... SUCCESS [8.638s]

[INFO] hadoop-mapreduce-client ........................... SUCCESS [0.135s]

[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [43.622s]

[INFO] hadoop-mapreduce-client-common .................... SUCCESS [36.329s]

[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [6.058s]

[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [20.058s]

[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [16.493s]

[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [11.685s]

[INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [3.222s]

[INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [12.656s]

[INFO] hadoop-mapreduce .................................. SUCCESS [8.060s]

[INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [8.994s]

[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [15.886s]

[INFO] Apache Hadoop Archives ............................ SUCCESS [6.659s]

[INFO] Apache Hadoop Rumen ............................... SUCCESS [15.722s]

[INFO] Apache Hadoop Gridmix ............................. SUCCESS [11.778s]

[INFO] Apache Hadoop Data Join ........................... SUCCESS [5.953s]

[INFO] Apache Hadoop Extras .............................. SUCCESS [6.414s]

[INFO] Apache Hadoop Pipes ............................... FAILURE [3.746s]

[INFO] Apache Hadoop OpenStack support ................... SKIPPED

[INFO] Apache Hadoop Client .............................. SKIPPED

[INFO] Apache Hadoop Mini-Cluster ........................ SKIPPED

[INFO] Apache Hadoop Scheduler Load Simulator ............ SKIPPED

[INFO] Apache Hadoop Tools Dist .......................... SKIPPED

[INFO] Apache Hadoop Tools ............................... SKIPPED

[INFO] Apache Hadoop Distribution ........................ SKIPPED

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 19:43.155s

[INFO] Finished at: Wed Jun 04 17:40:17 CST 2014

[INFO] Final Memory: 79M/239M

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-pipes: An Ant BuildException has occured: exec returned: 1

[ERROR] around Ant part ...<exec dir="/data/hadoop/hadoop-2.4.0-src/hadoop-tools/hadoop-pipes/target/native" executable="cmake" failonerror="true">... @ 5:123 in /data/hadoop/hadoop-2.4.0-src/hadoop-tools/hadoop-pipes/target/antrun/build-main.xml

[ERROR] -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

[ERROR]

[ERROR] After correcting the problems, you can resume the build with the command

根据网上提示( 下面需要再安装openssl-devel,安装命令yum install openssl-devel,此步不做的话会报如下错误

[exec] CMake Error at /usr/share/cmake/Modules/FindOpenSSL.cmake:66 (MESSAGE): 
[exec]   Could NOT find OpenSSL 
[exec] Call Stack (most recent call first): 
[exec]   CMakeLists.txt:20 (find_package) 
[exec] 
[exec] 
[exec] -- Configuring incomplete, errors occurred! 
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (make) on project hadoop-pipes: An Ant BuildException has occured: exec returned: 1 -> [Help 1] 
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 
[ERROR] Re-run Maven using the -X switch to enable full debug logging. 
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles: 
[ERROR] [Help 1] http://cwiki.apache.org/confluen ... oExecutionException 
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command 
[ERROR]   mvn <goals> -rf :hadoop-pipes

错误连接:http://f.dataguru.cn/thread-189176-1-1.html

原因是:在安装openssl-devel,少写一个l,重新安装一下

解决方法:重新安装openssl-devel

yum install openssl-devel

5.3.3 编译总结

1、 必须安装(yum install lzo-devel zlib-devel gcc autoconf automake libtool ncurses-devel openssl-devel)

2、 必须安装(protobuf,CMake)编译工具

3、 必须配置(ANT、MAVEN、FindBugs)

4、 将maven库指向开源中国,这样就可以加快编译速度,即加快下载依赖jar包速度

5、 编译出错需求详细观察出错日志,根据错误日志分析原因再结束百度和Google解决错误;

CentOS7x86_64编译hadoop2.4

  • 0

    开心

    开心

  • 0

    板砖

    板砖

  • 0

    感动

    感动

  • 0

    有用

    有用

  • 0

    疑问

    疑问

  • 0

    难过

    难过

  • 0

    无聊

    无聊

  • 0

    震惊

    震惊

编辑推荐
上次搭建hadoop1.2.1分布式集群,这次搭建hadoop2.4分布式集群,由于是在自己的笔记本上搭建集群,
4.3 Map类 创建Map类和map函数,map函数是org.apache.hadoop.mapreduce.Mapper类中的定义的,当处理
Android学习笔记之反编译工具介绍及下载(Class文件反编译,xml文件反编译整理)   最近在网上查
本文是关于实时 Java™ 的 系列文章(共 5 部分)的第二篇,考察了 Java 语言的本地代码编译所涉及
  最近在网上查了一些Android文件反编译工具,供大家参考。 1.CLASS文件反编译工具:XJAD 2.xml文
1.工具(请到网站搜索并自行下载): ①apktool(反编译:能得到图片资源与布局文件等) ②dex2jar
反编译工具 : 总结了一下 linux, windows, mac 上的版本, 一起放到 CSDN 上下载; -- CSDN 下载地址
反编译工具 : 总结了一下 linux, windows, mac 上的版本, 一起放到 CSDN 上下载; -- CSDN 下载地址
在编写ARX程序时,一个比较麻烦的地方在于,每次重新编译ARX项目时,需要先手动的卸载此前加载调试
以前写mapreduce,经常要做的同一件事情就是打包上传,每每做这件事情的时候都是痛苦的,由于之前ha
版权所有 IT知识库 CopyRight © 2009-2015 IT知识库 IT610.com , All Rights Reserved. 京ICP备09083238号