Zookeeper Quorum Architecture

 

1. High level Information about Zookeeper:

    - Distributed Key value store

    - Has voting mechanisms

    - Used by many big data tools

2. Absolutely necessary to have functional and up Zookeeper Quorum to run Apache Kafka.

    

Final Setup Architecture

 

Kafka Cluster setup is hard

1. To properly setup a Lafka Cluster, you need:

    - Zookeeper Cluster

    - Kafka Cluster

    - Proper replication factor

    - Multi Availability Zones Setup

    - Proper configurations(there are over 140 possible configs!)

Lesson 1 AWS Building Blocks Learning Objectives

 1.1 What is AWS?

AWS Official Definition

Amazon Web Services(AWS) is the world's most comprehensive and broadly adopted cloud (On-demand, Pay as you go, Network-accessible) platform, offering over 200 fully featured services(There is a service for almost everything, and you'll need to specialize!) from data centers globally.(Hundreds of data centers and millions of servers around the world!) Millions of customers - including the fastest-growing startups, largest enterprises, and leading government agencies - are using AWS to lower costs, become more agile, and innovate faster.(You can do these in ways not possible using on-premises data centers!)

1.2 The Shared Responsibility Model

1.3 The AWS Account

1.4 Demo: Introduction to the AWS Console

1.5 AWS Service Categories

1.6 AWS Icons and Diagrams

Module 1: AWS Overview

Lesson 1: AWS  Building Blocks

Lesson 2: AWS Global Infrastructure

Amazon Web Services(AWS) 3rd Edition

Module 1: AWS Overview

Module 2: AWS Identity and Access Management

Module 3: AWS Network Services

Module 4: AWS Compute Services

Module 5: AWS Storage Services

Module 6: AWS Database Services

Module 7: AWS High Availability Services

Module 8: AWS Analytics Services

Module 9: AWS Management Tools

Module 10: AWS Monitoring and Automation Services

Module 11: AWS Security Services

Module 12: AWS Developer Services

Module 13: AWS Biling and Cost Management

Module 14: Course Wrap-Up And Next Steps

Why should I care about topic config?

1. Brokers have defaults for all the topic configuration parameters

2. These parameters impact performance and topic behavior

3. Some topics may need different values than the defaults

    - Replication Factor

    - #of Partitions

    - Message size

    - Compression level

    - Log Cleanup Policy

    - Min Insync Replicas

    - Other configurations

4. A list of configuration can be found at:

https://kafka.apache.org/documentation/#brokerconfigs

.\kafka-topics.bat --bootstrap-server 127.0.0.1:2181 --list

PS C:\kafka_2.12-3.7.0> .\bin\windows\kafka-topics.bat --bootstrap-server 127.0.0.1:2181 --create --topic configured-topic --partitions 3 --replication-factor 1