What is Zookeeper

 1. Zookeeper provides multiple features for distributed applications:

    - Distributed configuration management

    - Self election / consensus building

    - Coordination and locks.

    - Key value store

2. Zookeeper is used in many distributed systems, such as Hadoop, Kafka, etc...

3. It's an Apache Project that's proven to be very stable and hasn't had major release in many years

4. 3.4.x is the stable channel

    3.5.x has been in development for manyu years, and it is still in beta

5. Zookeeper internal data structure is like a tree

- Each node is called a zNode

- Each zNode has a path

- zNode can be persistent or ephemeral

- Each zNode can store data

- No renaming of zNode

- Each zNode can be WATCHed for changes

Note on IPs and DNS *mostly AWS

 1. Your Zookeeper & Kafka must know their hostname and/or IP

2. These are not supposed to change over time, even after a reboot, otherwise your setup will be broken.

3. Options:

    - Use an Elastic Public IP == constat public IP

        you will be able to access your cluster from the outside (e.g. laptop)

    - Use a secondary ENI == constant private IP (this course)

        you will not be able to access your cluster from the outside

        you will only be able to access the cluster from within your network

    - Use DNS names (private or public) == no need to keep fixed IP

        public means you can access instances from outside

        private means you can't access instance from outside

Kafka Cluster Architecture

 

1. We will co-locate Zookeeper and Kafka for cost-saving purposes(not recommend for production deployments)

2. Knowledge is still applicable to production deployments

Zookeeper Quorum Architecture

 

1. High level Information about Zookeeper:

    - Distributed Key value store

    - Has voting mechanisms

    - Used by many big data tools

2. Absolutely necessary to have functional and up Zookeeper Quorum to run Apache Kafka.

    

Final Setup Architecture

 

Kafka Cluster setup is hard

1. To properly setup a Lafka Cluster, you need:

    - Zookeeper Cluster

    - Kafka Cluster

    - Proper replication factor

    - Multi Availability Zones Setup

    - Proper configurations(there are over 140 possible configs!)

Lesson 1 AWS Building Blocks Learning Objectives

 1.1 What is AWS?

AWS Official Definition

Amazon Web Services(AWS) is the world's most comprehensive and broadly adopted cloud (On-demand, Pay as you go, Network-accessible) platform, offering over 200 fully featured services(There is a service for almost everything, and you'll need to specialize!) from data centers globally.(Hundreds of data centers and millions of servers around the world!) Millions of customers - including the fastest-growing startups, largest enterprises, and leading government agencies - are using AWS to lower costs, become more agile, and innovate faster.(You can do these in ways not possible using on-premises data centers!)

1.2 The Shared Responsibility Model

1.3 The AWS Account

1.4 Demo: Introduction to the AWS Console

1.5 AWS Service Categories

1.6 AWS Icons and Diagrams