Note on IPs and DNS *mostly AWS

 1. Your Zookeeper & Kafka must know their hostname and/or IP

2. These are not supposed to change over time, even after a reboot, otherwise your setup will be broken.

3. Options:

    - Use an Elastic Public IP == constat public IP

        you will be able to access your cluster from the outside (e.g. laptop)

    - Use a secondary ENI == constant private IP (this course)

        you will not be able to access your cluster from the outside

        you will only be able to access the cluster from within your network

    - Use DNS names (private or public) == no need to keep fixed IP

        public means you can access instances from outside

        private means you can't access instance from outside

Kafka Cluster Architecture

 

1. We will co-locate Zookeeper and Kafka for cost-saving purposes(not recommend for production deployments)

2. Knowledge is still applicable to production deployments

Zookeeper Quorum Architecture

 

1. High level Information about Zookeeper:

    - Distributed Key value store

    - Has voting mechanisms

    - Used by many big data tools

2. Absolutely necessary to have functional and up Zookeeper Quorum to run Apache Kafka.

    

Final Setup Architecture

 

Kafka Cluster setup is hard

1. To properly setup a Lafka Cluster, you need:

    - Zookeeper Cluster

    - Kafka Cluster

    - Proper replication factor

    - Multi Availability Zones Setup

    - Proper configurations(there are over 140 possible configs!)

Lesson 1 AWS Building Blocks Learning Objectives

 1.1 What is AWS?

AWS Official Definition

Amazon Web Services(AWS) is the world's most comprehensive and broadly adopted cloud (On-demand, Pay as you go, Network-accessible) platform, offering over 200 fully featured services(There is a service for almost everything, and you'll need to specialize!) from data centers globally.(Hundreds of data centers and millions of servers around the world!) Millions of customers - including the fastest-growing startups, largest enterprises, and leading government agencies - are using AWS to lower costs, become more agile, and innovate faster.(You can do these in ways not possible using on-premises data centers!)

1.2 The Shared Responsibility Model

1.3 The AWS Account

1.4 Demo: Introduction to the AWS Console

1.5 AWS Service Categories

1.6 AWS Icons and Diagrams

Module 1: AWS Overview

Lesson 1: AWS  Building Blocks

Lesson 2: AWS Global Infrastructure