Hands-On: Web Tools AWS EC2 Machine

1. Start machine with Ubuntu

2. Install Docker and required packages

3. Open up ports

4. Try a Docker hello world

#!/bin/bash
sudo apt-get update

# Install packages to allow apt to use a repository over HTTPS:
sudo apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common

# Add Docker’s official GPG key:
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

# set up the stable repository.
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"

# install docker
sudo apt-get update
sudo apt-get install -y docker-ce docker-compose

# give ubuntu permissions to execute docker
sudo usermod -aG docker $(whoami)
# log out
exit
# log back in

# make sure docker is working
docker run hello-world

# Add hosts entries (mocking DNS) - put relevant IPs here
echo "172.31.9.1 kafka1
172.31.9.1 zookeeper1
172.31.19.230 kafka2
172.31.19.230 zookeeper2
172.31.35.20 kafka3
172.31.35.20 zookeeper3" | sudo tee --append /etc/hosts

Zookeeper in AWS

1. If you use private IPs, you may have the following error:

http://stackoverflow.com/questions/309409881/zookeeper-error-cannot-open-channel-to-x-at-election-address

2. Use Netflix Exhibitor

3. Reserve your instances if you know you will use them for over a year(decreased cost)

4. Or you can use Amazon EMR to provision a Zookeeper cluster, but you have less control

Zookeeper performance

1. Latency is key for Zookeeper, and any of these variables will affect it:

2. Fast disk(SSD)

3. No RAM Swap

4. Separate disk for snapshots and logs

5. High performance network (low latency)

6. Reasonable number of Zookeeper Servers

7. Islation of Zookeeper instances from other processes

Using Zookeeper Understanding the files created

1. Understanding the files created by Zookeeper on the filesystem

2. Besides myid, the rest of the files should remain untouched. They are managed by Zookeeper

3. /

    - myid: file representing the server id. That's how Zookeeper knows its identity

    - version-2/: folder that holds the zookeeper data

        - AcceptEpoch and CurrentEpoch: internal to Zookeeper

        - Log.X: Zookeeper data files

Hands-On: Using Zookeeper Four letter words

1. Examples of using 4LW (Four Letter Words)

2. https://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html#sczkCommands

# https://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html#sc_zkCommands
# conf
# New in 3.3.0: Print details about serving configuration.
echo "conf" | nc localhost 2181

# cons
# New in 3.3.0: List full connection/session details for all clients connected to this server. Includes information on numbers of packets received/sent, session id, operation latencies, last operation performed, etc...
echo "cons" | nc localhost 2181

# dump
# Lists the outstanding sessions and ephemeral nodes. This only works on the leader.
echo "dump" | nc localhost 2181

# envi
# Print details about serving environment
echo "envi" | nc localhost 2181

# ruok
# Tests if server is running in a non-error state. The server will respond with imok if it is running. Otherwise it will not respond at all.
echo "ruok" | nc localhost 2181

# srvr
# New in 3.3.0: Lists full details for the server.
echo "srvr" | nc localhost 2181

# stat
# Lists brief details for the server and connected clients.
echo "stat" | nc localhost 2181

# wchs
# New in 3.3.0: Lists brief information on watches for the server.
echo "wchs" | nc localhost 2181

#wchc
#New in 3.3.0: Lists detailed information on watches for the server, by session. This outputs a list of sessions(connections) with associated watches (paths). Note, depending on the number of watches this operation may be expensive (ie impact server performance), use it carefully.
echo "wchc" | nc localhost 2181

# wchp
# New in 3.3.0: Lists detailed information on watches for the server, by path. This outputs a list of paths (znodes) with associated sessions. Note, depending on the number of watches this operation may be expensive (ie impact server performance), use it carefully.
echo "wchp" | nc localhost 2181

# mntr
# New in 3.4.0: Outputs a list of variables that could be used for monitoring the health of the cluster.
echo "mntr" | nc localhost 2181

Hands On: Quorum Setup

1. Create an AMI (image) from  the existing machine

2. Create other 2 machines, and launch Zookeeper on them

3. Test that the Quorum is running and working

nano /home/ubuntu/kafka/config/zookeeper.properties

# the location to store the in-memory database snapshots and, unless specified otherwise, the transaction log of updates to the database.
dataDir=/data/zookeeper
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
# the basic time unit in milliseconds used by ZooKeeper. It is used to do heartbeats and the minimum session timeout will be twice the tickTime.
tickTime=2000
# The number of ticks that the initial synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# zoo servers
# these hostnames such as `zookeeper-1` come from the /etc/hosts file
server.1=zookeeper1:2888:3888
server.2=zookeeper2:2888:3888
server.3=zookeeper3:2888:3888

bin/zookeeper-server-start.sh config/zookeeper.properties

sudo mkdir -p /data/zookeeper
sudo chown -R ubuntu:ubuntu /data/
# declare the server's identity
echo "1" > /data/zookeeper/myid
# edit the zookeeper settings
rm /home/ubuntu/kafka/config/zookeeper.properties
nano /home/ubuntu/kafka/config/zookeeper.properties
# restart the zookeeper service
sudo service zookeeper stop
sudo service zookeeper start
# observe the logs - need to do this on every machine
cat /home/ubuntu/kafka/logs/zookeeper.out | head -100
nc -vz localhost 2181
nc -vz localhost 2888
nc -vz localhost 3888
echo "ruok" | nc localhost 2181 ; echo
echo "stat" | nc localhost 2181 ; echo
bin/zookeeper-shell.sh localhost:2181
# not happy
ls /

Hands-On: Using Zookeeper Command Line Interface

1. Create nodes, sub nodes, etc...

2. Get / Set data for a node

3. Watch a node

4. Delete a node

# start zookeeper
sudo service zookeeper start
# verify it's started
nc -vz localhost 2181

bin/zookeeper-shell.sh localhost 2181

# display help
help
# display root
ls /
create /my-node "foo"

ls /
get /my-node

set /my-node "new data"

create /my-node/deeper-node "bar"

ls /
ls /my-node
ls /my-node/deeper-node

get /my-node/deeper-node

rmr /my-node/deeper-node

rmr /my-node

ls /

# create a watcher
create /node-to-watch ""
get /node-to-watch true

set /node-to-watch "new data"

set /node-to-watch "whatever"