当前位置:首页 > 资讯 > 开源软件 > 正文

Flume ng1.6 + kafka 2.11 整合

发表于: 2015-12-25   作者:whitelover   来源:转载   浏览:
摘要: 第一步:安装ApacheFlume1.6;第二步:安装kafka;启动kafka步骤:启动zookeeper服务> bin/zookeeper-server-start.shconfig/zookeeper.properties&启动kafka> bin/kafka-server-start.shconfig/server.properties启动consumer(--from-beginning
  1. 第一步

    安装Apache Flume 1.6;

  2. 第二步

    安装kafka;

    启动kafka步骤:

    <1>启动zookeeper服务

    bin/zookeeper-server-start.sh config/zookeeper.properties &

    <2>启动kafka

    bin/kafka-server-start.sh config/server.properties

    <3>启动consumer(--from-beginning:此参数可以不要,表示启动之前接收到的消息也要消费 )

    bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

  3. 第三步

         配置flume 配置文件kafka-test.conf :

            #Name the components on this agent

            a1.sources = r1

            a1.sinks = k1

            a1.channels = c1

            #Describe/configure the source

            a1.sources.r1.type=netcat

            a1.sources.r1.bind=localhost

            a1.sources.r1.port=3333

            #Describe the sink ( kafka sink 配置)

            a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink

            a1.sinks.k1.topic = test

            a1.sinks.k1.brokerList =172.19.16.213:9092

            a1.sinks.k1.requiredAcks = 0

            a1.sinks.k1.batchSize = 20

            #kafka.producer.type=sync

            #kafka.partitioner.class=org.apache.flume.plugins.SinglePartition

            #Use a channel which buffers events in memory

            a1.channels.c1.type=memory

            a1.channels.c1.capacity=1000

            a1.channels.c1.transactionCapacity=100

            #Bind the source and sink to the channel

            a1.sources.r1.channels=c1

            a1.sinks.k1.channel=c1

    4.第四步 启动flume

    bin/flume-ng agent -c conf -f conf/kafka-test.conf --name a1 -Dflume.root.logger=INFO,console

 或

    bin/flume-ng agent -c conf -f conf/kafka-test.conf --name a1 

    5.第五步测试

    Flume ng1.6 + kafka 2.11 整合_第1张图片

    6.验证是否发送成功

    在启动consumer控制台查看接收到的消息。


Flume ng1.6 + kafka 2.11 整合

  • 0

    开心

    开心

  • 0

    板砖

    板砖

  • 0

    感动

    感动

  • 0

    有用

    有用

  • 0

    疑问

    疑问

  • 0

    难过

    难过

  • 0

    无聊

    无聊

  • 0

    震惊

    震惊

版权所有 IT知识库 CopyRight © 2009-2015 IT知识库 IT610.com , All Rights Reserved. 京ICP备09083238号