1. It's not easy to setup a cluster
2. You want to isolate each Zookeeper & Broker on separate servers
3. Monitoring needs to be implemented
4. Operations have to be mastered
5. You need a really good Kafka Admin
- Alternative: main different "Kafka as a Service" offerings on the web
- No operational burdens (upates, monitoring, setup, etc...)
1. You want multiple brokers in different data centers (racks) to distribute your load. You also want a cluster of at least 3 zookeeper
2. In AWS:
1. Four Common Kafka Use Cases:
Source => Kafka Producer API Kafka Connect Source
Kafka => Kafka Consumer, Producer API Kafka Streams
Kafka => Sink Consumer API Kafka Connect Sink
Kafka => App Consumer API
- Simplify and improve getting data in and out of Kafka
- Simplify transforming data wiythin Kafka without relying on external libs
- Programmers always want to import data from the same sources:
Databases, KDBC, Couchbase, GoldenGate, SAP HANA, Blockchain, Cassandra, DynamoDB, FTP, IOT, MongoDB, MQTT, RethinkDB, Salesforce, Solr, SQS, Twitter, etc...
- Programmers always want to store data in the same sinks:
S3, ElasticSearch, HDFS, KDBC, SAP HANA, DocumentDB, Cassandra, DynamoDB, HBase, MongoDB, Redis, Solr, Splunk, Twitter
- It is tough to achieve Fault Tolerance, IDempotence, Distribution, Ordering
- Other programmers may already have done a very good job!
1. (2013) Kafka 0.8.x:
- Topic replication, Log compaction
- Simplified producer client API
2. (Nov 2015) Kafka 0.9.x:
- Simplified high level consumer APIs, without Zookeeper dependency
- Added security (Encryption and Authentication)
- Kafka Connect APIs
3. (May 2016): Kafka 0.10.0:
- Kafka Streams APIS
4. (end 2016 - March 2017) Kafka 0.10.1, 0.10.2:
- Improved Connect API, Siungle Message Transforms API
1. Do you feel you're not the first person in the world to wirte a way to get data out of Twitter?
2. Do you feel like you;re not the first person in the world to send data from Kafka to PostgreSQL / ElasticSearch / MongoDb?
3. Additionally, the bugs you'll have, won't someone have fixed them already?
4. Kafka Connect is all about code & connectors re-use!
1. If the producer produces faster than the broker can take, the records will be buffered in memory
2. buffer.memory=33554432(32MB): the size of the send buffer
3. That buffer will fill up over time and fill back down when the throughput to the broker increases
4. If that buffer is full (all 32MB), then the .send() method will start to block (won't return right away)
5. max.block.ms=60000: the time the .send() will block until throwing a exception. Exceptions are basically thrown when
- The producer has filled up its buffer
- The broker is not accepting any new data
- 60 seconds has elapsed.
6. If you hit an exception hit that usually means your brokers aredown or overloaded as they can't respond to requests
