当前位置:首页 > 开发 > 开源软件 > 正文

hadoop2.5.2配置httpfs服务

发表于: 2015-05-08   作者:duguyiren3476   来源:转载   浏览次数:
摘要: hadoop2.5.2配置httpfs服务   httpfs hadoop hdfs   测试环境 ubuntu 14.04 单机 hadoop2.5.2 伪分布式 jdk1.7 作用 通过HttpFs你可以在浏览器里面管理HDFS上的文件,功能同hadoop shell相似 HttpFs还提供了一套REST

hadoop2.5.2配置httpfs服务

 

httpfs hadoop hdfs

 

测试环境

  • ubuntu 14.04 单机
  • hadoop2.5.2 伪分布式
  • jdk1.7

作用

  • 通过HttpFs你可以在浏览器里面管理HDFS上的文件,功能同hadoop shell相似
  • HttpFs还提供了一套REST 风格的API可以用来管理HDFS

缺点

  • 网络安全性
  • 需要配置acl

修改core-site.xml

 
    <property>
        <name>hadoop.proxyuser.root.hosts</name>
        <value>localhost</value>
    </property>
    <property>
        <name>hadoop.proxyuser.root.groups</name>
        <value>*</value>
    </property>

添加上述两个配置hadoop.proxyuser.root.hosts允许通过httpfs方式访问hdfs的主机名或者域名;hadoop.proxyuser.root.groups允许访问的客户端的用户组

配置httpfs环境变量

 
export CATALINA_BASE=/opt/work/hadoop/share/hadoop/httpfs/tomcat

临时配置,也可以在配置文件配置环境变量

重启hadoop集群

 
root@localhost:/opt/nfs#/opt/work/hadoop/sbin//stop-all.sh 
root@localhost:/opt/nfs#/opt/work/hadoop/sbin//start-all.sh 

启动httpfs服务

 
root@localhost:/opt/git/hadoop_dev/hdfsToInfoBright# /opt/work/hadoop/sbin/httpfs.sh start

Setting HTTPFS_HOME:          /opt/work/hadoop
Setting HTTPFS_CONFIG:        /opt/work/hadoop/etc/hadoop
Sourcing:                    /opt/work/hadoop/etc/hadoop/httpfs-env.sh
Setting HTTPFS_LOG:           /opt/work/hadoop/logs
Setting HTTPFS_TEMP:           /opt/work/hadoop/temp
Setting HTTPFS_HTTP_PORT:     14000
Setting HTTPFS_ADMIN_PORT:     14001
Setting HTTPFS_HTTP_HOSTNAME: localhost
Setting HTTPFS_SSL_ENABLED: false
Setting HTTPFS_SSL_KEYSTORE_FILE:     /root/.keystore
Setting HTTPFS_SSL_KEYSTORE_PASS:     password
Using   CATALINA_BASE:       /opt/work/hadoop/share/hadoop/httpfs/tomcat/
Setting HTTPFS_CATALINA_HOME:       /opt/work/hadoop/share/hadoop/httpfs/tomcat/
Setting CATALINA_OUT:        /opt/work/hadoop/logs/httpfs-catalina.out
Setting CATALINA_PID:        /tmp/httpfs.pid

Using   CATALINA_OPTS:       
Adding to CATALINA_OPTS:     -Dhttpfs.home.dir=/opt/work/hadoop -Dhttpfs.config.dir=/opt/work/hadoop/etc/hadoop -Dhttpfs.log.dir=/opt/work/hadoop/logs -Dhttpfs.temp.dir=/opt/work/hadoop/temp -Dhttpfs.admin.port=14001 -Dhttpfs.http.port=14000 -Dhttpfs.http.hostname=localhost -Dhttpfs.ssl.enabled=false -Dhttpfs.ssl.keystore.file=/root/.keystore -Dhttpfs.ssl.keystore.pass=password
Using CATALINA_BASE:   /opt/work/hadoop/share/hadoop/httpfs/tomcat/
Using CATALINA_HOME:   /opt/work/hadoop/share/hadoop/httpfs/tomcat
Using CATALINA_TMPDIR: /opt/work/hadoop/share/hadoop/httpfs/tomcat//temp
Using JRE_HOME:        /usr/local/jdk1.7.0
Using CLASSPATH:       /opt/work/hadoop/share/hadoop/httpfs/tomcat//bin/tomcat-juli.jar:/opt/work/hadoop/share/hadoop/httpfs/tomcat/bin/bootstrap.jar
Using CATALINA_PID:    /tmp/httpfs.pid
Existing PID file found during start.
Removing/clearing stale PID file.
root@localhost:/opt/git/hadoop_dev/hdfsToInfoBright# 

并查看启动日志,并确定环境变量的正确CATALINA_BASE的值

浏览器访问httpfs

http://localhost:14000/ 
页面显示:HttpFs service, service base URL at /webhdfs/v1. 根目录为:/webhdfs/v1

curl访问httpfs

 
root@localhost:/opt/nfs# curl -i -X PUT -T /opt/test.json "http://localhost:14000/webhdfs/v1/tmp/test.json?op=CREATE&data=true&user.name=root" -H "Content-Type:application/octet-stream"

文件上传到hdfs 
http访问文件: 
http://localhost:14000/webhdfs/v1/tmp/test.json?user.name=root&op=open 
即可下载test.json文件

考虑

  • httpfs的安全性
  • httpfs的性能:httpfs是一个tomcat启动的web服务,当数据文件非常大,上传下载的性能将骤减
  • httpfs的并发访问:tomcat的并发问题

参考文献

hadoop2.5.2配置httpfs服务

  • 0

    开心

    开心

  • 0

    板砖

    板砖

  • 0

    感动

    感动

  • 0

    有用

    有用

  • 0

    疑问

    疑问

  • 0

    难过

    难过

  • 0

    无聊

    无聊

  • 0

    震惊

    震惊

编辑推荐
配置免登录 Ø创建或者修改密钥目录权限 假设本机的IP为“192.168.1.100”,配置免登录的操作步骤如
Apache Hadoop HttpFS服务提供了通过HTTP访问HDFS,HttpFS自带REST HTTP API, 支持所有HDFS文件系
环境:CentOS6.6 64bit 、JDK1.7.0_72 一、安装与编译有关的包 [root@hadoop4 ~]# yum install auto
一.hadoop集群环境配置(完整配置参考上一篇博客) 1.在master节点上(master)修改hdfs-site.xml加上以
一、环境 转载请出至出处:http://eksliang.iteye.com/blog/2223784 准备3台虚拟机,安装Centos 64-
一.hadoop集群环境配置(完整配置参考上一篇博客) 1.在master节点上(master)修改hdfs-site.xml加上以
一、环境 转载请出至出处:http://eksliang.iteye.com/blog/2223784 准备3台虚拟机,安装Centos 64-
环境: 三台CentOS6.6虚拟机,机器名分别为hadoop4、hadoop5、hadoop6,通过桥接连网,互相可以ping
Error Message: "The program can't start because MSCVR100.dll is missing from your computer. T
转载请出自出处:http://eksliang.iteye.com/blog/2191493 一、下载hadoop的的部署文件 我这里是使
版权所有 IT知识库 CopyRight © 2009-2015 IT知识库 IT610.com , All Rights Reserved. 京ICP备09083238号