1. In the case of 3 brokers:
- N-1 brokers can be down, if N is your default topic replication factor.
- Ex: If N = 3, then two brokers can be down
- Producer and consumer requests are spread out between your different machines
- Data is spread out between brokers, which means less disk space is used per broker.
- You can have a cluster
1. In the case of only 1 broker:
- If the broker is restarted, the Kafka cluster is down
- The maximum replication factor for topics is 1.
- All producer and consumer requests go to the same unique broker
- You can only scale vertically (by increasing the instance size and restarting)
2. It's extremely high risk, and only useful for development purposes
1. Brokers holds topic partitions
2. Brokers receive and serve data
3. Brokers are the unit of parallelism of a Kafka cluster
4. Brokers are the essence of the "distributed" aspect of Kafka
1. You can build your own using the 4 Letter Words
2. Or use one of the following:
- Netflix Exhibitor (heavily recommended but tedious setup):
https://github.com/soabase/exhibitor
- Zookeeper UI (web):
https://github.com/DeemOpen/zkui
- Zookeeper GUI(desktop) - Windows binaries available:
https://github.com/echoma/zkui
- ZK-web(not updated since July 2016):
https://github.com/qiuziafei/zk-web
- ZooNavigator(promising new project):
https://github.com/elkozmon/zoonavigator
zoonavigator-docker-compose.yml
version: '2'
services:
# https://github.com/elkozmon/zoonavigator
zoonavigator:
image: elkozmon/zoonavigator:latest
container_name: zoonavigator
network_mode: host
environment:
HTTP_PORT: 8001
restart: always
nano zoonavigator-docker-compose.yml
# Make sure port 8001 is opened on the instance security group
# copy the zookeeper/zoonavigator-docker-compose.yml file
# run it
docker-compose -f zoonavigator-docker-compose.yml up -d
docker ps
curl localhost:8001
1. Start machine with Ubuntu
2. Install Docker and required packages
3. Open up ports
4. Try a Docker hello world
#!/bin/bash
sudo apt-get update
# Install packages to allow apt to use a repository over HTTPS:
sudo apt-get install -y \
apt-transport-https \
ca-certificates \
curl \
software-properties-common
# Add Docker’s official GPG key:
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
# set up the stable repository.
sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
# install docker
sudo apt-get update
sudo apt-get install -y docker-ce docker-compose
# give ubuntu permissions to execute docker
sudo usermod -aG docker $(whoami)
# log out
exit
# log back in
# make sure docker is working
docker run hello-world
# Add hosts entries (mocking DNS) - put relevant IPs here
echo "172.31.9.1 kafka1
172.31.9.1 zookeeper1
172.31.19.230 kafka2
172.31.19.230 zookeeper2
172.31.35.20 kafka3
172.31.35.20 zookeeper3" | sudo tee --append /etc/hosts
1. If you use private IPs, you may have the following error:
http://stackoverflow.com/questions/309409881/zookeeper-error-cannot-open-channel-to-x-at-election-address
2. Use Netflix Exhibitor
3. Reserve your instances if you know you will use them for over a year(decreased cost)
4. Or you can use Amazon EMR to provision a Zookeeper cluster, but you have less control
1. Latency is key for Zookeeper, and any of these variables will affect it:
2. Fast disk(SSD)
3. No RAM Swap
4. Separate disk for snapshots and logs
5. High performance network (low latency)
6. Reasonable number of Zookeeper Servers
7. Islation of Zookeeper instances from other processes
