Big Data for Architects

Video description

Get to grips with big data technologies and work on real-world big data projects confidently

About This Video

Get a holistic picture of the Big Data ecosystem
Become an expert in choosing Big Data technology as per the requirements
Get ready to build end-to-end Big Data batch and streaming pipelines

In Detail

Do you want a guide that will help you to pick the right Big Data technology for your project? Or do you want to get a solid understanding of the Big …

Big Data for Architects

Video description

Get to grips with big data technologies and work on real-world big data projects confidently

About This Video

Get a holistic picture of the Big Data ecosystem
Become an expert in choosing Big Data technology as per the requirements
Get ready to build end-to-end Big Data batch and streaming pipelines

In Detail

Do you want a guide that will help you to pick the right Big Data technology for your project? Or do you want to get a solid understanding of the Big Data architecture and pipelines? This course will help you out.

After highlighting the course structure and learning objectives, the course will take you through the steps needed for setting up the environment. Next, you will understand the Big Data logical architecture, study the evolution of Big Data technologies, and explore Big Data pipelines. Moving along, you will become familiar with ingestion frameworks, such as Kafka, Flume, Nifi, and Sqoop. Next, you will learn about key storage frameworks, such as HDFS, HBase, Kudu, and Cassandra. Finally, you will go through the various data formats and uncover key data processing and data analysis frameworks.

By the end of this course, you will have a good understanding of the Big Data architecture and technologies and will have developed the skills to build real-world Big Data pipelines.

Who this book is for

If you are a software engineer, who is looking to build Big Data pipelines or planning to appear for certifications such as CCA175 or CCA159, this video course is for you. A basic understanding of Big Data is needed to get started with this course.

Publisher resources

Download Example Code

Chapter 1 : Introduction

Course Structure and Approach

Course Pre-Requisites

Course Audience

About the Author

Chapter 2 : Setting Up the Environment

Setting up a Google Cloud Account

Creating a Dataproc Cluster

Google Cloud Platform (GCP) Account Best Practices

Chapter 3 : Holistic View of Architectures and Pipelines

Big Data Logical Architecture

Evolution of Big Data Technologies

Key Big Data Architectures

Typical Big Data Batch Pipeline

Typical Big Data Streaming Pipeline

Example 01: Big Data Streaming Pipeline

Example 02: Big Data Streaming Pipeline

Chapter 4 : Key Ingestion/Dataflow Frameworks

Factors to Consider while Comparing Ingestion Frameworks

Kafka Versus Flume

NiFi Versus Kafka

Sqoop Versus Flume

Sqoop Versus Kafka Connect

Installing NiFi

Installing Kafka

Hands-on Kafka and NiFi Integration Background

Integrating Kafka and NiFi

Chapter 5 : Key Storage Frameworks

Factors to Consider while Comparing Storage Frameworks

Hadoop Distributed File System (HDFS) Versus HBase

HBase Versus Kudu

Hadoop Distributed File System (HDFS) Versus Kudu

HBase Versus Cassandra

Chapter 6 : Data formats

Text Versus Binary

Interoperability

Row-Oriented Versus Column-Oriented

Splittable Formats

Schema Evolution

Comparing Data Formats

Installing Sqoop on Dataproc Cluster

Hands-on Big Data Batch Pipeline Using the Avro Format

Chapter 7 : Key Data Processing Frameworks

Factors to Consider while Comparing Processing Frameworks

MapReduce (MR) Versus Spark Logical Architecture

MapReduce (MR) Versus Spark Performance

Spark Versus Tez

Spark Versus Flink

Kafka Streams Versus Spark Streaming

Spark 2.x Streaming Versus Spark 1.x Streaming

Spark Core Versus Spark Structured Query Language (SQL)

Integrating Kafka and Spark Streaming

Chapter 8 : Key Data Analysis Frameworks

Factors to Consider while Comparing Analysis Frameworks

Hive Versus Impala

Hive Versus Pig

Hive Versus Spark Structured Query Language (SQL)

Hive Versus Hive Live Long and Process (LLAP) Versus Impala

Hive Versus KSQL

KSQL Versus KSQLDB

Hands-On KSQL

Writing to a Stream and Table Using KSQL

Streaming Extract, Transform, Load (ETL) Pipeline Background

Building a Scalable Extract, Transform, Load (ETL) Pipeline with Kafka Connect - Part 1

Building a Scalable Extract, Transform, Load (ETL) Pipeline with Kafka Connect - Part 2

Chapter 9 : Delta Lake

Delta Architecture

Why Delta Lake

Challenges with Delta Lake

Delta Lake Demo

Chapter 10 : Additional Material

Solr Versus Elasticsearch

Cloudera Search Versus Solr

Oozie Versus Airflow

KSQL Versus KStreams

Chapter 11 : Summary

Conclusion

Start your Free Trial

Self paced

Go to the Course
We have partnered with providers to bring you collection of courses, When you buy through links on our site, we may earn an affiliate commission from provider.