Can I run Kafka Streams Application on the same machine as of Kafka Broker? -


i have kafka streams application takes data few topics , joins data , puts in topic.

kafka configuration:

5 kafka brokers kafka topics - 15 partitions , 3 replication factor.  

note: running kafka streams applications on same machines kafka brokers running.

few millions of records consumed/produced every hour. whenever take kafka broker down, goes rebalancing , takes approx. 30 minutes or more rebalancing , many times kills many of kafka streams processes.

it technically possible run kafka streams application on same servers broker. but not recommended. both need share same resources , end contention.

whenever take kafka broker down, goes rebalancing

not sure why happening. version of kafka or streams api using? if on broker 0.10.1+ highly recommend upgrade streams application 0.11 (note, can without broker upgrade).

depending on details of issue phasing, standbytask might long rebalance times. can simple configure parameter num.standby.replica = 1 enable standbytasks.


Comments

Popular posts from this blog

angular - Ionic slides - dynamically add slides before and after -

minify - Minimizing css files -

Add a dynamic header in angular 2 http provider -