In just 10 hours, learn how to handle real-time data feed using Apache Kafka open-source messaging
About This Video
In Detail
Do you want to learn Apache Kafka to store and process multiple, nonstop streams of information …
In just 10 hours, learn how to handle real-time data feed using Apache Kafka open-source messaging
About This Video
In Detail
Do you want to learn Apache Kafka to store and process multiple, nonstop streams of information faster? Or do you want to prepare for the Apache Kafka certification exam? This course is an ideal resource to learn Apache Kafka and to reach your goals.
The course starts with an introduction to big data and Apache Kafka. Next, you will get into the details of a Kafka producer and consumer and grasp the techniques to tune the performance of Kafka operations. Moving along, you will explore the Kafka architecture and learn how to administer Kafka effectively. Next, you will learn to monitor server, infrastructure, and application and understand how the Schema Registry works in Kafka. Later, you will get to grips with Kafka Streams and Connectors and learn the process of integrating Kafka with Apache Storm, Spark, and Flume. Towards the end, you will learn how to perform various admin tasks using the admin clients and uncover useful tips to secure Kafka.
By the end of this course, you will have developed the Kafka skills needed to handle real-time data feed and to pass the Apache Kafka certification exam.
Who this book is for
If you are into analytics or research or are a software developer, test engineer, or a project manager, who is looking to learn Apache Kafka and achieve an Apache Kafka certification, this course is for you. A basic understanding of big data is needed to get started with this course.
Chapter 1 : Course Introduction
Introduction
Chapter 2 : Getting Started with Big data and Apache Kafka
Learning Objectives
Data Evolution over Time
Big Data Overview
Big Data Analytics
Messaging System
Introduction and Features of Kafka
Kafka Use Cases
Kafka Terminologies
Kafka Components
Kafka Architecture
Kafka Clusters
Kafka Industry Use Cases
Demo - Installing Zookeeper
Demo - Installing Kafka
Demo – Setting up a Single Node and Single Broker Cluster
Demo – Setting up a Multi-node Single Broker Cluster
Key Takeaways
Chapter 3 : Kafka Producers
Learning Objective
Overview of a Kafka Producer and Its Architecture
Kafka Producer Configuration
Kafka Producer Optional Configuration
Kafka Producer Configuration Objects
Demo - Creating a Kafka Producer
Sending Messages
Serializers and Custom Serializers
Demo - Creating a Custom Serializer
Serializers Challenges and Serializing Using Apache Avro
Demo - Serializing Using Apache Avro
Partitions
Demo – Setting up a Custom Partition
Key Takeaway
Chapter 4 : Kafka Consumers
Learning Objective
Overview of Kafka Consumers
Consumer Groups
Partition Rebalance and Creating a Kafka Consumer
Poll Loop and Its Functioning
Configuring a Kafka Consumer - Part 1
Configuring a Kafka Consumer - Part 2
Demo - Creating a Kafka Consumer
Commit and Offset
Ways of Committing Offset - Automatic Offset
Ways of Committing Offset - Commit Current Offset
Ways of Committing Offset - Asynchronous Commit
Ways of Committing Offset - Combining Synchronous and Asynchronous Commits
Ways of Committing Offset - Commit Specified Offset
Rebalance Listeners
Consuming Records with Specific Offset
Deserializers
Demo - Creating and Using Custom Deserializer
Key Takeaways
Chapter 5 : Kafka Operations and Performance Tuning
Learning Objective
Kafka Internals Overview
Replication and Replica Types
Preferred Ladder, Request, and Request Processing
Types of Requests
Partition Allocation, File Management, and Segments
File Format, Index, and Compaction
Kafka Reliability and Reliability Methods
Broker Configuration for Replication
Producer in a Reliable System
Consumer in a Reliable System
Key Takeaways
Chapter 6 : Kafka Cluster Architectures and Administering Kafka
Learning Objective
Cluster Mirroring
Multi-cluster Architecture - Hub, Spokes, and Active Architecture – Part 1
Multi-cluster Architecture – Active/Stand-by Architecture and Stretch Clusters Part – 2
Configuring MirrorMaker – Part 1
Configuring MirrorMaker – Part 2
Demo - Setting up MirrorMaker
Administering Kafka - Topic Operations
Administering Kafka - Consumer Group Operations
Dynamic Configuration Changes
Partition Management
Console Producer Tool
Console Consumer Tool
Key Takeaways
Chapter 7 : Monitoring Kafka and Schema Registry
Learning Objective
Monitoring and Its Importance
Server or Infrastructure Monitoring and Application Monitoring
Monitoring Kafka
Kafka Broker Metrics – Under-replicated Partitions – Part 1
Kafka Broker Metrics – Others– Part 2
Topic and Partition Specific Metrics
Logging and Client Monitoring
Producer and Consumer Metrics
Quotas and Lag Monitoring
Monitoring Dashboard
Demo - Setting up an Open Source Health Monitor
Kafka Schema Registry
Kafka Components and Its Architecture
Kafka Schema Registry - Internal working and Use cases
Working of the Kafka Schema Registry
Demo – Using the Kafka Schema Registry with Kafka
Key Takeaway
Chapter 8 : Kafka Streams and Kafka Connectors
Learning Objective
Kafka Streams Overview
Kafka Streams
Kafka Streams Architecture and Working
Kafka Streams Components
Kafka Streams Architecture Tasks, Threading Model and Local State Store
Kafka Streams Architecture - Record Buffer
Memory Management and Streaming Data Pipeline
Kafka Streams Domain Specific Language (DSL)
KStream Operations
KTable Operations
KTable Operations
Aggregation and Windowing
Processor Topology and Stream Processor
Stream and Processor Application Programming Interfaces (APIs)
Processor Application Programming Interfaces (APIs) and Creating Topology
Kafka Connectors
Stand-alone and Sink Connector Configuration
Running Kafka Connect
Kafka Connector Distributed Mode
Hypertext Transfer Protocol (HTTP) Representational State Transfer (REST) Interface
Demo - Kafka Connector
Demo - Creating an Application Using Kafka Streams
Key Takeaway
Chapter 9 : Integration of Kafka with Storm
Learning Objective
Apache Storm
Real-time Analytics
Apache Storm Architecture
Apache Storm Components
Apache Storm Topology Part - 1
Apache Storm Topology Part - 2
Kafka Spout
Integration of Apache Storm and Kafka
Demo - Simple Stand-alone Application Using Kafka and Storm
Key Takeaway
Chapter 10 : Kafka Integration with Spark and Flume
Learning Objective
Introduction to Spark
Spark Components
Resilient Distributed Dataset
Resilient Distributed Dataset (RDD) Operations - Transformation - Map, FlatMap, and Filter
RDD Operations - Transformation - Join, Distinct, First, and Take
Resilient Distributed Dataset (RDD) Operations - Actions
Data Sets and SparkSession
Data Sets and Spark Session Operations
Spark Stream
Integrating Spark with Kafka
Demo - Running a Small Stand-alone Application in Spark with Kafka
Flume
Flume Connectors
Configuring Flume Kafka to Hadoop Distributed File System (HDFS)
Demo - Creating a Flume Agent and Sending Data from Kafka to Hadoop Distributed File System (HDFS)
Key Takeaways
Chapter 11 : Admin Client and Securing Kafka
Learning Objective
AdminClient
Demo - Performing Various Admin Tasks Using AdminClient
Kafka Security
Kafka Security Components
Simple Authentication and Security Layer (SASL)
Configuring Simple Authentication and Security Layer (SASL) in Kafka
Securing Using Access Control Lists (ACLs)
Key Takeaway