Producers Acks Deep Dive acks = 1 (leader acks)

 1. Leader response is requested, but replication is not a guarantee

(happens in the background)

2. If an ack is not received, the produceder may retry

3. If the leader broker goes offline but replicas haven't replicated the data yet, we have a data loss.

Producers Acks Deep Dive acks = 0 (no acks)

1. No response is requested

2. If the broker goes offline or an exception happens, we won't know and will lose data

3. Useful for data where it's okay to potentially lose messages:

    - Metrics collection

    - Log collection

Kafka Command Line Interface 101

 kafka-topics

C:\kafka_2.12-3.7.0>.\bin\windows\kafka-topics.bat --bootstrap-server localhost:9092,localhost:9093,localhost:9094 --create --topic mytopic --partitions 2 --replication-factor 1

Created topic mytopic.

Start Kafka

 https://kafka.apache.org/ download

mkdir

C:\kafka_2.12-3.7.0\data\zookeeper

setting path: bin

C:\kafka_2.12-3.7.0\config\zookeeper.properties edit 

dataDir=C:/kafka_2.12-3.7.0/data/zookeeper

C:\kafka_2.12-3.7.0 zookeeper-server-start.bat config\zookeeper.properties

C:\kafka_2.12-3.7.0>bin\windows\zookeeper-server-start.bat config\zookeeper.properties

[2024-06-23 08:53:35,580] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)

C:\kafka_2.12-3.7.0\config\server.properties edit 

log.dirs=C:/kafka_2.12-3.7.0/data/kafka

C:\kafka_2.12-3.7.0>.\bin\windows\kafka-server-start.bat config\server.properties

[2024-06-23 09:00:46,925] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

Theory Roundup We've looked at all the Kafka concepts

 

Kafka Guarantees

- Messages are appended to a topic-partition in the order they are sent

- Consumers read messages in the order stored in a topic-partition

- With a replication factor of N, producers and consumers can tolerate up to N-I brokers being down

- This is why a replication factor of 3 is a good idea:

    - Allows for on broker to be taken down for maintenance

    - Allows for another broker to be taken down unexpectedly

- As long as the number of partitions remains constant for a topic(no new partitions), the same key will always go to the same partition

Zookeeper

- Zookeeper manages brokers (keeps a list of them)

- Zookeeper helps in performing leader election for partitions

- Zookeeper sends notifications to Kafka in case of changes (e.g. new topic, broker dies, broker comes up, delete topics, etc...)

- Kafka can't work without Zookeeper

- Zookeeper by design operates with an odd number of servers (3, 5, 7)

- Zookeeper has a leader (handle writes) the rest of the servers are followers (handle reads)

- (Zokeeper does NOT store consumer offsets with Kafka > v0.10)