Apache Kafka Series - Learn Apache Kafka for Beginners v3

Video description

Understand Apache Kafka 3.0 and its configuration and apply the knowledge in real-time use cases.

About This Video

Find out why Apache Kafka is an excellent choice of distributed data-streaming platform
Learn major CLIs: kafka-topics, kafka-console-producer, kafka-console-consumer, kafka-consumer-groups, kafka-configs
Get a solid grip on Apache Kafka by understanding various real-world examples

In Detail

The high throughput …

Apache Kafka Series - Learn Apache Kafka for Beginners v3

Video description

Understand Apache Kafka 3.0 and its configuration and apply the knowledge in real-time use cases.

About This Video

Find out why Apache Kafka is an excellent choice of distributed data-streaming platform
Learn major CLIs: kafka-topics, kafka-console-producer, kafka-console-consumer, kafka-consumer-groups, kafka-configs
Get a solid grip on Apache Kafka by understanding various real-world examples

In Detail

The high throughput and low latency of Apache Kafka have made it one of the leading distributed data-streaming enterprise technologies. It is used by many Fortune 500 companies such as Netflix, Airbnb, Uber, Walmart, and LinkedIn. If you want to develop the Apache Kafka skills to stream data easily, then this course is for you.

The course starts by explaining the architecture of the Apache Kafka ecosystem, before going on to cover the core concepts of Kafka such as topics, partitions, brokers, replicas, producers, consumers, and more. Next, you will use native Kafka binaries to launch your own Kafka cluster on Windows, Mac OS X, and Linux.

As you advance, you will get hands-on experience with the Kafka command-line interface (CLI), along with understanding how to create producers and consumers in Java to interact with Kafka. Next, a real-world project using Wikimedia as a source of data for a producer and OpenSearch as a sink for our consumers. Moving on, you will get to grips with advanced APIs such as Kafka Connect and Kafka Streams and perform a case study on the real-world applications of Kafka. Finally, you will get an overview of advanced Kafka for administrators and understand advanced topic configurations.

By the end of this course, you will be well-versed with how Apache Kafka 3.0 plays an important role in data-streaming applications.

Audience

This course is for developers who want to learn the fundamentals of Apache Kafka, software architects who want to understand how Kafka fits into their solution architecture, or anyone who is looking to understand how Apache Kafka works as a distributed system. Basic knowledge of Java and the Linux command line will be beneficial to effectively understanding the concepts covered in the course.

Publisher resources

Download Example Code

Chapter 1 : Kafka Introduction

Course Introduction

Apache Kafka in Five Minutes

Course Objectives

Chapter 2 : Kafka Theory

Topics, Partitions, and Offsets

Producers and Message Keys

Consumers and Deserialization

Consumer Groups and Consumer Offsets

Brokers and Topics

Topic Replication

Producer Acknowledgments and Topic Durability

Zookeeper

Kafka Kraft – Removing Zookeeper

Theory Roundup

Chapter 3 : Starting Kafka

Important: Starting Kafka

Starting Kafka with Conduktor – Multi Platform

Mac OS X - Download Kafka and Set Up PATH

Mac OS X - Start Zookeeper and Kafka

Mac OS X - Using Brew

Linux - Download and Set Up Kafka in PATH

Linux - Start Zookeeper and Kafka

Windows WSL2 - Download Kafka and PATH Setup

Windows WSL2 - Start Zookeeper and Kafka

Windows Non-WSL2 - Start Zookeeper and Kafka

Chapter 4 : Starting Kafka without Zookeeper

Mac OS X - Start Kafka in Kraft Mode

Linux - Start Kafka in Kraft Mode

Windows WSL2 - Start Kafka in Kraft Mode

Chapter 5 : Command Line Interface (CLI) 101

Command Line Interface (CLI) Introduction

Kafka Topics Command Line Interface (CLI)

Kafka Console Producer Command Line Interface (CLI)

Kafka Console Consumer Command Line Interface (CLI)

Kafka Consumers in a Group

Kafka Consumer Groups Command Line Interface (CLI)

Resetting Offsets

Chapter 6 : Kafka UI

Conduktor - Demo

Chapter 7 : Kafka Java Programming 101

Kafka SDK List

Creating Kafka Project

Java Producer

Java Producer Callback

Java Producer with Keys

Java Consumer

Java Consumer – Graceful Shutdown

Java Consumer Inside a Consumer Group

Java Consumer Incremental Cooperative Rebalance and Static Group Membership

Java Consumer Incremental Cooperative Rebalance - Practice

Java Consumer Auto Offset Commit Behavior

Programming - Advanced Tutorials

Chapter 8 : Kafka Real-World Project

Real-World Project Overview

Chapter 9 : Kafka Wikimedia Producer and Advanced Configurations

Wikimedia Producer Project Setup

Wikimedia Producer Implementation

Wikimedia Producer Run

Wikimedia Producer - Producer Config Intros

Producer Acknowledgements Deep Dive

Producer Retries

Idempotent Producer

Safe Kafka Producer Settings

Wikimedia Producer Safe Producer Implementation

Kafka Message Compression

linger.ms and batch.size Producer Settings

Wikimedia Producer High Throughput Implementation

Producer Default Partitioner and Sticky Partitioner

[Advanced] max.block.ms and buffer.memory

Chapter 10 : OpenSearch Consumer and Advanced Consumer Configurations

OpenSearch Consumer - Project Overview

OpenSearch Consumer - Project Setup

Setting Up OpenSearch on Docker

Setting Up OpenSearch on the Cloud

OpenSearch 101

OpenSearch Consumer Implementation - Part 1

OpenSearch Consumer Implementation Part 2

Consumer Delivery Semantics

OpenSearch Consumer Implementation Part 3 - Idempotence

Consumer Offsets Commit Strategies

OpenSearch Consumer Implementation Part 4 - Delivery Semantics

OpenSearch Consumer Implementation Part 5 - Batching Data

Consumer Offset Reset Behavior

OpenSearch Consumer Implementation Part 6 - Replaying Data

Consumer Internal Threads

Consumer Replica Fetching - Rack Awareness

Chapter 11 : Kafka Extended APIs for Developers

Kafka Extended APIs - Overview

Kafka Connect Introduction

Kafka Connect Wikimedia and ElasticSearch Hands-On

Kafka Streams Introduction

Kafka Streams Hands-On

Kafka Schema Registry Introduction

Kafka Schema Registry Hands-On

Which Kafka API Should I Use?

Chapter 12 : Real-World Insights and Case Studies (Big Data / Fast Data)

Choosing Partition Count and Replication Factor

Kafka Topics Naming Convention

Case Study – MovieFlix

Case Study – GetTaxi

Case Study – MySocialMedia

Case Study – MyBank

Case Study - Big Data Ingestion

Case Study - Logging and Metrics Aggregation

Chapter 13 : Kafka in the Enterprise for Admins

Kafka Cluster Setup High-Level Architecture Overview

Kafka Monitoring and Operations

Kafka Security

Kafka Multi-Cluster and MirrorMaker

Advertised Listeners: Kafka Client and Server Communication Protocol

Chapter 14 : Advanced Topics Configurations

Changing a Topic Configuration

Segment and Indexes

Log Cleanup Policies

Log Cleanup Delete

Log Compaction Theory

Log Compaction Practice

Unclean Leader Election

Large Messages in Kafka

Chapter 15 : Next Steps

What’s Next?

Start your Free Trial

Self paced

Go to the Course
We have partnered with providers to bring you collection of courses, When you buy through links on our site, we may earn an affiliate commission from provider.