Factors impacting Kafka performance Network

1. Latency is key in Kafka

    - Ensure your Kafka instances are your Zookeeper instances are geographically close!!!

    - Do not put one broker in Europe and the other broker in the US

    - Having two brokers live on the same rack is good for performance, but a big risk if the rack goes down.

2. Network bandwidth is key in Kafka

    - Network will be your bottleneck.

    - Make sure you have enough bandwidth to handle manyu connections, and TCP requests.

    - Make sure your network is high performance

3. Monitor network usage to understand when it becomes a bottleneck

Factors impacting Kafka performance

1. Reads are done sequentially (as in not randomly), therefore make sure you should a disk type that corresponds well to the requirements

2. Format your drives as XFS(easiest, no tuning required)

3. If read/write throughput is your bottleneck

    - it is possible to mount multiple disks in parallel for Kafka

    - The config is log.dirs=/disk1/kafka-logs, /disk2/kafka-logs,/disk3/kafka-logs...

4. Kafka performance is constant with regards to the amount of data stored in Kafka.

    - Make sure you expire data fast enough (default is one week)

    - Make sure you monitor disk performance

Hands On: Demonstrating Kafka resiliency

1. We create a topic with 3 as a replication factor

2. We will start producing data consistently to a topic

3. We will read that data from a topic

4. We will kill one Kafka instance

5. We will kill another Kafka instance

6. We will kill the last Kafka instance

#!/bin/bash

# create a topic with replication factor of 3
bin/kafka-topics.sh --zookeeper zookeeper1:2181,zookeeper2:2181,zookeeper3:2181/kafka --create --topic fourth_topic --replication-factor 3 --partitions 3

# generate 10KB of random data
base64 /dev/urandom | head -c 10000 | egrep -ao "\w" | tr -d '\n' > file10KB.txt

# in a new shell: start a continuous random producer
bin/kafka-producer-perf-test.sh --topic fourth_topic --num-records 10000 --throughput 10 --payload-file file10KB.txt --producer-props acks=1 bootstrap.servers=kafka1:9092,kafka2:9092,kafka3:9092 --payload-delimiter A

# in a new shell: start a consumer
bin/kafka-console-consumer.sh --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --topic fourth_topic

# kill one kafka server - all should be fine
# kill another kafka server -
# kill the last server

# start back the servers one by one


# ERRORS:
# PRODUCER:
# org.apache.kafka.common.errors.TimeoutException: Expiring 137 record(s) for fourth_topic-0: 30024 ms has passed since batch creation plus linger time
# [2017-05-25 10:24:23,784] WARN Error while fetching metadata with correlation id 1086 : {fourth_topic=INVALID_REPLICATION_FACTOR} (org.apache.kafka.clients.NetworkClient)
# org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.
# [2017-05-25 10:24:23,850] WARN Received unknown topic or partition error in produce request on partition fourth_topic-0. The topic/partition may not exist or the user may not have Describe access to it (org.apache.kafka.clients.producer.internals.Sender)
# [2017-05-25 10:24:23,914] WARN Error while fetching metadata with correlation id 1092 : {fourth_topic=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
# org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.

# CONSUMER:
# [2017-05-25 10:24:23,798] WARN Error while fetching metadata with correlation id 3431 : {fourth_topic=INVALID_REPLICATION_FACTOR} (org.apache.kafka.clients.NetworkClient)
# [2017-05-25 10:24:24,081] WARN Auto-commit of offsets {fourth_topic-0=OffsetAndMetadata{offset=3948, metadata=''}} failed for group console-consumer-25246: Offset commit failed with a retriable exception. You should retry committing offsets. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
# [2017-05-25 10:24:24,231] WARN Received unknown topic or partition error in fetch for partition fourth_topic-0. The topic/partition may not exist or the user may not have Describe access to it (org.apache.kafka.clients.consumer.internals.Fetcher)

Hands On: Cluster Management with Kafka Manager

1. Setup security groups to access our web tools instance

2. Run Kafka anager using Kocker Compose

3. Demo of Kafka Manager

#!/bin/bash

# make sure you open port 9000 on the security group

# make sure you can access the zookeeper endpoints
nc -vz zookeeper1 2181
nc -vz zookeeper2 2181
nc -vz zookeeper3 2181

# make sure you can access the kafka endpoints
nc -vz kafka1 9092
nc -vz kafka2 9092
nc -vz kafka3 9092

# copy the kafka-manager-docker-compose.yml file
nano kafka-manager-docker-compose.yml

# launch it
docker-compose -f kafka-manager-docker-compose.yml up -d

version: '2'

services:
  # https://github.com/yahoo/kafka-manager
  kafka-manager:
    image: qnib/plain-kafka-manager
    network_mode: host
    environment:
      ZOOKEEPER_HOSTS: "zookeeper1:2181,zookeeper2:2181,zookeeper3:2181"
      APPLICATION_SECRET: change_me_please
    restart: always

So what do I set for advertised.listeners?

1. If you clients are on your private network, set either:

    - the internal private IP

    - the internal private DNS hotname

2. Your clients should be able to resolve the internal IP or hostname

3. If your clients are on a public network, set either:

    - The external public IP

    - The external public DNS hostname pointing to the public IP

4. Your clients must be able to resolve the public DNS

What if I use the public IP... But after a reboot Kafka public IP changed!

1. Assume the IP of your server has changed:

    - FROM 34.56.78.90 => TO 34.56.78.12

What if I use the public IP?

 1. Advertised hostname is the most important setting of Kafka