我有以下结构:
zookeeper: 3.4.12
kafka: kafka_2.11-1.1.0
server1: zookeeper + kafka
server2: zookeeper + kafka
server3: zookeeper + kafka
通过 kafka-topics shell 脚本创建了复制因子为 3 且分区为 3 的主题。
./kafka-topics.sh --create --zookeeper localhost:2181 --topic test-flow --partitions 3 --replication-factor 3
并使用本地消费者组。当领导没问题时,它工作得很好。
./kafka-topics.sh --describe --zookeeper localhost:2181 --topic test-flow
Topic:test-flow PartitionCount:3 ReplicationFactor:3 Configs:
Topic: test-flow Partition: 0 Leader: 3 Replicas: 3,2,1 Isr: 3,2,1
Topic: test-flow Partition: 1 Leader: 1 Replicas: 1,3,2 Isr: 1,3,2
Topic: test-flow Partition: 2 Leader: 2 Replicas: 2,1,3 Isr: 2,1,3
消费者日志
Received FindCoordinator response ClientResponse(receivedTimeMs=1529508772673, latencyMs=217, disconnected=false, requestHeader=RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=1, clientId=consumer-1, correlationId=0), responseBody=FindCoordinatorResponse(throttleTimeMs=0, errorMessage='null', error=NONE, node=myserver3:9092 (id: 3 rack: null)))
但是如果领导者宕机了 - 我会在消费者中收到错误(systemctl stop kafka):
节点 3 不可用。好的
./kafka-topics.sh --describe --zookeeper localhost:2181 --topic test-flow
Topic:test-flow PartitionCount:3 ReplicationFactor:3 Configs:
Topic: test-flow Partition: 0 Leader: 2 Replicas: 3,2,1 Isr: 2,1
Topic: test-flow Partition: 1 Leader: 1 Replicas: 1,3,2 Isr: 1,2
Topic: test-flow Partition: 2 Leader: 2 Replicas: 2,1,3 Isr: 2,1
消费者日志
Received FindCoordinator response
ClientResponse(receivedTimeMs=1529507314193, latencyMs=36,
disconnected=false,
requestHeader=RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=1,
clientId=consumer-1, correlationId=149),
responseBody=FindCoordinatorResponse(throttleTimeMs=0,
errorMessage='null', error=COORDINATOR_NOT_AVAILABLE, node=:-1 (id: -1
rack: null)))
- Group coordinator lookup failed: The coordinator is not available.
- Coordinator discovery failed, refreshing metadata
在领导者宕机或与另一个消费者组重新连接之前,消费者无法连接。
不明白为什么会发生?
消费者应该重新平衡到另一个经纪人,但事实并非如此。