Safe producer Summary & Demo

 Kafka < 0.11

- acks = all (producer level)

    - Ensures data is properly replicated before an ack is received

- min.insync.replicas=2 (broker/topic level)

    - Ensures two brokers in ISR at least have the data after an ack

- retries=MAX_INT (producer level)

    - Ensures transient errors are retried indefinitely

- max.in.flight.requests.per.connection=1 (producer level)

    - Ensures onlyu one request is tried at any time, preventing message re-ordering in case of retries

Kafka >= 0.11

- enable.idempotence=true (producer level) + min.insync.replicas=2 (broker/topic level)

    - Implies acks=all, retries=MAX_INT, max.in.flight.requests.per.connection=5(default)

    - while keeping ordering guarantees and improving performance!

- Running a "safe producer" might impact throughput an latency, always test for your use case