Learn in detail about Apache Kafka platform, which is used by giants such as Netflix and LinkedIn. A hands-on approach to build Kafka consumer applications using Java to reinforce your learning.
About This Video
In Detail
This course will walk you through the Kafka concepts step by step. We will …
Learn in detail about Apache Kafka platform, which is used by giants such as Netflix and LinkedIn. A hands-on approach to build Kafka consumer applications using Java to reinforce your learning.
About This Video
In Detail
This course will walk you through the Kafka concepts step by step. We will explore the Kafka internals from a theoretical perspective followed by hands-on exploring the Kafka internals. Then we will learn about building the Kafka Producer to produce messages into Kafka using the producer API. This is followed by learning the different configurations that guarantee message delivery into Kafka and building Kafka Consumer to read messages from Kafka using the consumer API.
In the next section, we will learn about consumer groups and consumer rebalance, consumer offsets, and the different strategies that are available for the consumer to commit the offsets.
Next, we will learn about consumer rebalance listeners to perform some clean-up before partition revocation or some initialization tasks during partition assignment. Then learn about the different techniques from the consumer to seek a different offset.
And finally, we will code and learn how to build custom Serializer/Deserializer and integrate it into our Kafka Producer/Consumer.
By the end of the course, you will have a complete understanding of Apache Kafka and will be able to build Kafka Consumer applications using Java.
Who this book is for
This course is designed for Kafka beginners and advanced level. Anyone interested in building Java applications using producer and consumer API or interested in learning advanced Kafka Producer and Consumer operations, then this course is for you.
Chapter 1 : Getting Started with the Course
Course Introduction and Objectives
Prerequisites
Chapter 2 : Getting Started with Kafka
Introduction to Kafka
Kafka Terminologies and Client APIs
Chapter 3 : Download and Install Kafka
Download and Install Kafka
Chapter 4 : Understanding Kafka Components and its Internals - (Theory + Hands-On)
Kafka Topics and Partitions - Theory
Set Up a ZooKeeper/Kafka Broker in Local
Create Topic, Produce and Consume Messages Using the CLI
Produce and Consume Messages with Key
Consumer Offsets
Consumer Groups
Commit Log and Retention Policy
Kafka as a Distributed Streaming System
Setting up a Kafka Cluster in Local with 3 Kafka Brokers
How Kafka Cluster Distributes the Client Requests? Leader/Follower
How Kafka Handles Data Loss? Replication and In-Sync-Replica(ISR)
Fault Tolerance and Robustness in Kafka
Chapter 5 : Kafka Producer API - Hands-On
Project Setup
Sending ProducerRecord Using Producer API
Build Kafka Producer - Send Messages Synchronously
Add Logging Using LogBack
Sending Messages Asynchronously
Send Messages with Key
Programmatic Interface to Produce Messages
Kafka Producer API - Behind the Scenes
Chapter 6 : Kafka Producer API - Guaranteed Message Delivery Configurations
Kafka Producer API Configurations
Configuring Acks and min.insync.replicas – Hands-On
Configuring retries and retry.backoff.ms – Hands-On
Chapter 7 : Kafka Consumer API – Hands-On
Project Setup
Consuming Messages using Consumer API - poll() loop
Build Kafka Consumer Using the Consumer API
auto.offset.reset - Earliest Versus Latest
Chapter 8 : Consumer Groups and Consumer Rebalance – Hands-On
Scaling the Consumer Using Consumer Groups
Consumer Rebalance
Consumer Rebalance and max.poll.interval.ms
Chapter 9 : Consumer Offsets - Default and Manual Offset Management – Hands-On
Committing Offsets - Default Offset Commit Strategy
Manually Committing Offsets
Committing Offsets Synchronously - Using commitSync()
Committing Offsets Asynchronously - Using commitAsync()
Committing Specific Offsets - using CommitSync()
Chapter 10 : Consumer Rebalance Listeners – Hands-On
Introduction to Rebalance Listeners
Implement ConsumerRebalanceListener in Kafka Consumer
Committing Offsets Using ConsumerRebalanceListener
Chapter 11 : Kafka Consumer - seekToBeginning(), seekToEnd(), and seek() – Hands-On
Introduction to seekToBeginning() or seekToEnd()
seekToBeginning() and seekToEnd() – Hands-On
Introduction to seek()
Seek to a Specific Offset Using seek() – Hands-On
Chapter 12 : Custom Serializer and Deserializers in Kafka – Hands-On
Why Custom Kafka Serializer/Deserializer?
Build a Custom Serializer in Kafka Producer Using Jackson ObjectMapper
Build Kafka Producer Using Custom Serializer
Build a Custom DeSerializer in Kafka Producer Using Jackson ObjectMapper
Build Kafka Consumer Using Custom Serializer
Publish Custom Objects Using StringSerializer and Jackson ObjectMapper
Consume Custom Objects using StringDeserializer and Jackson ObjectMapper