Hands On: Cluster Management with Kafka Manager

1. Setup security groups to access our web tools instance

2. Run Kafka anager using Kocker Compose

3. Demo of Kafka Manager

#!/bin/bash

# make sure you open port 9000 on the security group

# make sure you can access the zookeeper endpoints
nc -vz zookeeper1 2181
nc -vz zookeeper2 2181
nc -vz zookeeper3 2181

# make sure you can access the kafka endpoints
nc -vz kafka1 9092
nc -vz kafka2 9092
nc -vz kafka3 9092

# copy the kafka-manager-docker-compose.yml file
nano kafka-manager-docker-compose.yml

# launch it
docker-compose -f kafka-manager-docker-compose.yml up -d

version: '2'

services:
  # https://github.com/yahoo/kafka-manager
  kafka-manager:
    image: qnib/plain-kafka-manager
    network_mode: host
    environment:
      ZOOKEEPER_HOSTS: "zookeeper1:2181,zookeeper2:2181,zookeeper3:2181"
      APPLICATION_SECRET: change_me_please
    restart: always

So what do I set for advertised.listeners?

1. If you clients are on your private network, set either:

    - the internal private IP

    - the internal private DNS hotname

2. Your clients should be able to resolve the internal IP or hostname

3. If your clients are on a public network, set either:

    - The external public IP

    - The external public DNS hostname pointing to the public IP

4. Your clients must be able to resolve the public DNS

What if I use the public IP... But after a reboot Kafka public IP changed!

1. Assume the IP of your server has changed:

    - FROM 34.56.78.90 => TO 34.56.78.12

What if I use the public IP?

 1. Advertised hostname is the most important setting of Kafka

But what if I put localhost? It works on my machine!

 1. Advertised hostname is the most important setting of Kafka

Understanding communications between Client and with Kafka

1. Advertised listeners is the most important setting of Kafka

Can I connect from the outside? e.g. my computer?

1. Zookeeper:

    - If you used Elastic private IPs(like in the tutorial), YES - demo

    - If you used Elastic public IPs                           YES

2. Kafka   

    - If you used Elastic private IPs(like in this tutorial), NO - demo

    - If you used Elastic public IPs                            YES