当前位置:首页 > 开发 > 开源软件 > 正文

【Kafka四】Kakfa伪分布式安装

发表于: 2015-02-22   作者:bit1129   来源:转载   浏览:
摘要: 在http://bit1129.iteye.com/blog/2174791一文中,实现了单Kafka服务器的安装,在Kafka中,每个Kafka服务器称为一个broker。本文简单介绍下,在单机环境下Kafka的伪分布式安装和测试验证   1. 安装步骤   Kafka伪分布式安装的思路跟Zookeeper的伪分布式安装思路完全一样,不过比Zookeeper稍微简单些(不

在http://bit1129.iteye.com/blog/2174791一文中,实现了单Kafka服务器的安装,在Kafka中,每个Kafka服务器称为一个broker。本文简单介绍下,在单机环境下Kafka的伪分布式安装和测试验证

 

1. 安装步骤

 

Kafka伪分布式安装的思路跟Zookeeper的伪分布式安装思路完全一样,不过比Zookeeper稍微简单些(不需要创建myid文件),主要是针对每个Kafka服务器配置一个单独的server.properties,三个服务器分别使用server.properties,server.1.properties, server.2.properties

 

cp server.properties server.1.properties
cp server.properties server.2.properties

 

修改server.1.properties和server.2.properties,主要有三个属性需要修改

 

broker.id=1
port=9093
log.dirs=/tmp/kafka-logs-1
 

port指的是Kakfa服务器监听的端口

 

启动三个Kafka:
bin/kafka-server-start.sh server.properties

bin/kafka-server-start.sh server.1.properties

bin/kafka-server-start.sh server.2.properties

 

 

2. Kafka脚本常用配置参数

2.1 kafka-console-consumer.sh

--from-beginning                        If the consumer does not already have an established offset to consume from, start with the earliest message present in the log rather than the latest message. 

--topic <topic>                           The topic id to consume on

--zookeeper <urls>                    REQUIRED: The connection string for the zookeeper connection in the form host:port. Multiple URLS can be given to allow fail-over.

--group <gid>                            The group id to consume on. (default: console-consumer-37803)

在consumer端,不需要指定broke-list,而是通过zookeeper和topic找到所有的持有topic消息的broker

 

2.2 kafka-console-producer.sh

--topic <topic>                         REQUIRED: The topic id to produce  messages to.

--broker-list <broker-list>        REQUIRED: The broker list string in the form HOST1:PORT1,HOST2:PORT2.

 

2.3 kafka-topic.sh

--create                                Create a new topic.

--describe                              List details for the given topics.

--list                                  List all available topics.

--partitions <Integer: # of partitions> The number of partitions for the topic being created or altered (WARNING:   If partitions are increased for a  topic that has a key, the partition logic or ordering of the messages will be affected)

--replication-factor <Integer: replication factor> The replication factor for each partition in the topic being created

--zookeeper <urls>                    REQUIRED: The connection string for the zookeeper connection in the form host:port. Multiple URLS can be given to allow fail-over.

--topic <topic>                         The topic to be create, alter or describe. Can also accept a regular expression except for --create option

 

3. 伪机群测试

测试前,先总结有哪些测试点

目前想到的是,Partition有个leader的概念,leader partition是什么意思?干什么用的?

 

3.1 创建Topic

./kafka-topics.sh --create  --topic  topic_p10_r3 --partitions 10 --replication-factor 3  --zookeeper localhost:2181

 创建一个Topic,10个Partition,副本数为3,也就是说,每个broker上的每个分区,在其它节点都有副本,因为每个节点都有10个节点的数据

 

3.2 每个broker创建的目录

当创建完Topic后,每个Topic都会在Kakfa的配置目录下(比如/tmp/kafka-logs,建立相应的目录和文件)

topic_p10_r3-0

topic_p10_r3-1

----

topic_p10_r3-9

其中每个目录下面都有两个文件: 00000000000000000000.index  00000000000000000000.log

 

3.3 Topic的详细信息

 

./kafka-topics.sh --describe --topic topic_p10_r3  --zookeeper localhost:2181

得到的结果如下:

 

Topic:topic_p10_r3	PartitionCount:10	ReplicationFactor:3	Configs:
	Topic: topic_p10_r3	Partition: 0	Leader: 2	Replicas: 2,0,1	Isr: 2,0,1
	Topic: topic_p10_r3	Partition: 1	Leader: 0	Replicas: 0,1,2	Isr: 0,1,2
	Topic: topic_p10_r3	Partition: 2	Leader: 1	Replicas: 1,2,0	Isr: 1,2,0
	Topic: topic_p10_r3	Partition: 3	Leader: 2	Replicas: 2,1,0	Isr: 2,1,0
	Topic: topic_p10_r3	Partition: 4	Leader: 0	Replicas: 0,2,1	Isr: 0,2,1
	Topic: topic_p10_r3	Partition: 5	Leader: 1	Replicas: 1,0,2	Isr: 1,0,2
	Topic: topic_p10_r3	Partition: 6	Leader: 2	Replicas: 2,0,1	Isr: 2,0,1
	Topic: topic_p10_r3	Partition: 7	Leader: 0	Replicas: 0,1,2	Isr: 0,1,2
	Topic: topic_p10_r3	Partition: 8	Leader: 1	Replicas: 1,2,0	Isr: 1,2,0
	Topic: topic_p10_r3	Partition: 9	Leader: 2	Replicas: 2,1,0	Isr: 2,1,0

具体的含义是:

Here is an explanation of output. The first line gives a summary of all the partitions, each additional line gives information about one partition

  • "leader" is the node responsible for all reads and writes for the given partition. Each node will be the leader for a randomly selected portion of the partitions.
  • "replicas" is the list of nodes that replicate the log for this partition regardless of whether they are the leader or even if they are currently alive.
  • "isr" is the set of "in-sync" replicas. This is the subset of the replicas list that is currently alive and caught-up to the leader.

3.4 问题: 如果副本数为1,是否表示每个partition在集群中只有1份(也就是说每个partition只会存在于一个broker上),那么leader自然就表示这个partition就在leader所指的broker上了?

 

建立包含10个分区,同时只有一个副本的topic

 

./kafka-topics.sh --create  --topic  topic_p10_r1 --partitions 10 --replication-factor 1  --zookeeper localhost:2181

详细信息:

 

[hadoop@hadoop bin]$ ./kafka-topics.sh --describe --topic topic_p10_r1  --zookeeper localhost:2181
Topic:topic_p10_r1	PartitionCount:10	ReplicationFactor:1	Configs:
	Topic: topic_p10_r1	Partition: 0	Leader: 1	Replicas: 1	Isr: 1
	Topic: topic_p10_r1	Partition: 1	Leader: 2	Replicas: 2	Isr: 2
	Topic: topic_p10_r1	Partition: 2	Leader: 0	Replicas: 0	Isr: 0
	Topic: topic_p10_r1	Partition: 3	Leader: 1	Replicas: 1	Isr: 1
	Topic: topic_p10_r1	Partition: 4	Leader: 2	Replicas: 2	Isr: 2
	Topic: topic_p10_r1	Partition: 5	Leader: 0	Replicas: 0	Isr: 0
	Topic: topic_p10_r1	Partition: 6	Leader: 1	Replicas: 1	Isr: 1
	Topic: topic_p10_r1	Partition: 7	Leader: 2	Replicas: 2	Isr: 2
	Topic: topic_p10_r1	Partition: 8	Leader: 0	Replicas: 0	Isr: 0
	Topic: topic_p10_r1	Partition: 9	Leader: 1	Replicas: 1	Isr: 1

 可见理解不错,每个partition有不同的leader,Leader所在的broker同时也是Replicas所在的broker(ID号一样)

因此可以理解,

1. 每个partition副本集都有一个leader

2. leader指的是partition副本集中的leader,它负责读写,然后负责将数据复制到其它的broker上。

3.一个Topic的所有partition会比较均匀的分布到多个broker上

 

3.5 broker挂了,Kafka的容错机制

在上面已经建立了两个Topic,一个是10个分区3个副本, 一个是10个分区1个副本。此时,假如有一个broker挂了,看看这两个Topic的容错如何?

通过jps命令可以看到有三个Kafka进程。

通过ps -ef|grep server.2.properties可以找到brokerId为2的Kakfa进程,使用kill -9将其干掉。干掉的时候,console开始刷屏,异常信息一样,都是:

 

 

[2015-02-23 02:14:00,037] WARN Reconnect due to socket error: null (kafka.consumer.SimpleConsumer)
[2015-02-23 02:14:00,039] ERROR [ReplicaFetcherThread-0-2], Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 4325; ClientId: ReplicaFetcherThread-0-2; ReplicaId: 1; MaxWait: 500 ms; MinBytes: 1 bytes; RequestInfo: [topic_p10_r3,3] -> PartitionFetchInfo(0,1048576),[topic_p10_r3,9] -> PartitionFetchInfo(0,1048576),[topic_p10_r3,6] -> PartitionFetchInfo(0,1048576),[topic_p10_r3,0] -> PartitionFetchInfo(0,1048576) (kafka.server.ReplicaFetcherThread)
java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.connect0(Native Method)
	at sun.nio.ch.Net.connect(Net.java:465)
	at sun.nio.ch.Net.connect(Net.java:457)
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
	at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
	at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:44)
	at kafka.consumer.SimpleConsumer.reconnect(SimpleConsumer.scala:57)
	at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:79)
	at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:109)
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:109)
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:109)
	at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:108)
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:108)
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:108)
	at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
	at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:107)
	at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
	at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
	at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
[2015-02-23 02:14:00,040] WARN Reconnect due to socket error: null (kafka.consumer.SimpleConsumer)

 3,9,6,0正是topic_p10_r3上broker2作为leader的partition,可见Kafka要做Leader移交,看看此时的topic_p10_r3和topic_p10_r1的情况

topic_p10_r3(Partition切换到其它Leader上了。。。Rplicas还有3,。。。)

[hadoop@hadoop bin]$ ./kafka-topics.sh --describe --topic topic_p10_r3  --zookeeper localhost:2181
Topic:topic_p10_r3	PartitionCount:10	ReplicationFactor:3	Configs:
	Topic: topic_p10_r3	Partition: 0	Leader: 0	Replicas: 2,0,1	Isr: 0,1
	Topic: topic_p10_r3	Partition: 1	Leader: 0	Replicas: 0,1,2	Isr: 0,1
	Topic: topic_p10_r3	Partition: 2	Leader: 1	Replicas: 1,2,0	Isr: 1,0
	Topic: topic_p10_r3	Partition: 3	Leader: 1	Replicas: 2,1,0	Isr: 1,0
	Topic: topic_p10_r3	Partition: 4	Leader: 0	Replicas: 0,2,1	Isr: 0,1
	Topic: topic_p10_r3	Partition: 5	Leader: 1	Replicas: 1,0,2	Isr: 1,0
	Topic: topic_p10_r3	Partition: 6	Leader: 0	Replicas: 2,0,1	Isr: 0,1
	Topic: topic_p10_r3	Partition: 7	Leader: 0	Replicas: 0,1,2	Isr: 0,1
	Topic: topic_p10_r3	Partition: 8	Leader: 1	Replicas: 1,2,0	Isr: 1,0
	Topic: topic_p10_r3	Partition: 9	Leader: 1	Replicas: 2,1,0	Isr: 1,0

 

topic_p10_r1:没有切换,但是Leader是-1了。。

 

[hadoop@hadoop bin]$ ./kafka-topics.sh --describe --topic topic_p10_r1  --zookeeper localhost:2181
Topic:topic_p10_r1	PartitionCount:10	ReplicationFactor:1	Configs:
	Topic: topic_p10_r1	Partition: 0	Leader: 1	Replicas: 1	Isr: 1
	Topic: topic_p10_r1	Partition: 1	Leader: -1	Replicas: 2	Isr: 
	Topic: topic_p10_r1	Partition: 2	Leader: 0	Replicas: 0	Isr: 0
	Topic: topic_p10_r1	Partition: 3	Leader: 1	Replicas: 1	Isr: 1
	Topic: topic_p10_r1	Partition: 4	Leader: -1	Replicas: 2	Isr: 
	Topic: topic_p10_r1	Partition: 5	Leader: 0	Replicas: 0	Isr: 0
	Topic: topic_p10_r1	Partition: 6	Leader: 1	Replicas: 1	Isr: 1
	Topic: topic_p10_r1	Partition: 7	Leader: -1	Replicas: 2	Isr: 
	Topic: topic_p10_r1	Partition: 8	Leader: 0	Replicas: 0	Isr: 0
	Topic: topic_p10_r1	Partition: 9	Leader: 1	Replicas: 1	Isr: 1

 

重启broker 2得到结果如下:(对于topic_p10_r3,leader没有变化,即每个Partition都有自己的Leader,新加入的broker只能follower;而topic_p10_r1,则会选出Leader)

 

[hadoop@hadoop bin]$ ./kafka-topics.sh --describe --topic topic_p10_r3  --zookeeper localhost:2181
Topic:topic_p10_r3	PartitionCount:10	ReplicationFactor:3	Configs:
	Topic: topic_p10_r3	Partition: 0	Leader: 0	Replicas: 2,0,1	Isr: 0,1,2
	Topic: topic_p10_r3	Partition: 1	Leader: 0	Replicas: 0,1,2	Isr: 0,1,2
	Topic: topic_p10_r3	Partition: 2	Leader: 1	Replicas: 1,2,0	Isr: 1,0,2
	Topic: topic_p10_r3	Partition: 3	Leader: 1	Replicas: 2,1,0	Isr: 1,0,2
	Topic: topic_p10_r3	Partition: 4	Leader: 0	Replicas: 0,2,1	Isr: 0,1,2
	Topic: topic_p10_r3	Partition: 5	Leader: 1	Replicas: 1,0,2	Isr: 1,0,2
	Topic: topic_p10_r3	Partition: 6	Leader: 0	Replicas: 2,0,1	Isr: 0,1,2
	Topic: topic_p10_r3	Partition: 7	Leader: 0	Replicas: 0,1,2	Isr: 0,1,2
	Topic: topic_p10_r3	Partition: 8	Leader: 1	Replicas: 1,2,0	Isr: 1,0,2
	Topic: topic_p10_r3	Partition: 9	Leader: 1	Replicas: 2,1,0	Isr: 1,0,2
[hadoop@hadoop bin]$ ./kafka-topics.sh --describe --topic topic_p10_r1  --zookeeper localhost:2181
Topic:topic_p10_r1	PartitionCount:10	ReplicationFactor:1	Configs:
	Topic: topic_p10_r1	Partition: 0	Leader: 1	Replicas: 1	Isr: 1
	Topic: topic_p10_r1	Partition: 1	Leader: 2	Replicas: 2	Isr: 2
	Topic: topic_p10_r1	Partition: 2	Leader: 0	Replicas: 0	Isr: 0
	Topic: topic_p10_r1	Partition: 3	Leader: 1	Replicas: 1	Isr: 1
	Topic: topic_p10_r1	Partition: 4	Leader: 2	Replicas: 2	Isr: 2
	Topic: topic_p10_r1	Partition: 5	Leader: 0	Replicas: 0	Isr: 0
	Topic: topic_p10_r1	Partition: 6	Leader: 1	Replicas: 1	Isr: 1
	Topic: topic_p10_r1	Partition: 7	Leader: 2	Replicas: 2	Isr: 2
	Topic: topic_p10_r1	Partition: 8	Leader: 0	Replicas: 0	Isr: 0
	Topic: topic_p10_r1	Partition: 9	Leader: 1	Replicas: 1	Isr: 1

 

 

 

 

 

 

【Kafka四】Kakfa伪分布式安装

  • 0

    开心

    开心

  • 0

    板砖

    板砖

  • 0

    感动

    感动

  • 0

    有用

    有用

  • 0

    疑问

    疑问

  • 0

    难过

    难过

  • 0

    无聊

    无聊

  • 0

    震惊

    震惊

编辑推荐
安装环境 Hadoop版本:1.1.2 虚拟机:virtualbox 4.3.8.0 服务器:Ubuntu service 13.10 x64 java:
0 注意: hadoop2中 不识别 _ 如果你在hadoop2的配置文件中出现_ 会报错,这个作者在下面案例中使用
Hadoop1.2.1 单机模式安装 Hadoop组件依赖图(从下往上看) 安装步骤: 详细步骤:   设置ssh自动登
由于本人笔记本配置太低,安装一个虚拟机就卡的一戳一戳的,,,,所以只能安装伪分布式的集群 真正
1、下载scala2.11.5版本,下载地址为:http://www.scala-lang.org/download/2.11.5.html 2、安装和
转载请出自出处:http://eksliang.iteye.com/blog/2191493 一、下载hadoop的的部署文件 我这里是使
Hadoop2.3.0伪分布式安装笔记 安装环境:ubuntu 13.10, jdk1.7.0_51, hadoop 2.3.0 一、在ubuntu下
转载请出自出处:http://eksliang.iteye.com/blog/2191493 一、下载hadoop的的部署文件 我这里是使
参考连接: http://www.cnblogs.com/xia520pi/archive/2012/05/16/2503949.html 1.0环境说明 a) VMw
1、下载scala2.11.5版本,下载地址为:http://www.scala-lang.org/download/2.11.5.html 2、安装和
版权所有 IT知识库 CopyRight © 2009-2015 IT知识库 IT610.com , All Rights Reserved. 京ICP备09083238号