Sharpen your architectural skills by understanding challenges in the main areas of distributed systems: storage, computation, messaging, timing, and consensus. You’ll learn how to develop highly scalable big data applications using Apache Accumulo, to model and design an agile data warehouse, and to use Elasticsearch to search, aggregate, analyze, and scale large volume datastores. You’ll also learn how to identify insecurities …
Sharpen your architectural skills by understanding challenges in the main areas of distributed systems: storage, computation, messaging, timing, and consensus. You’ll learn how to develop highly scalable big data applications using Apache Accumulo, to model and design an agile data warehouse, and to use Elasticsearch to search, aggregate, analyze, and scale large volume datastores. You’ll also learn how to identify insecurities in your big data cluster, and to secure them using MIT Kerberos, authentication with Active Directory, and authorization.
Welcome
What Distributed Systems Are, and Why They Exist
Read Replication
Sharding
Consistent Hashing
CAP Theorem
Distributed Transactions
Distributed Computation Introduction
Map Reduce
Hadoop
Spark
Storm
Lambda Architecture
Synchronization
Network Time Protocol
Vector Clocks
Distributed Consensus: Paxos
Messaging Introduction
Kafka
Zookeeper
Wrap-Up
Getting Started
About The Course
About The Author
What Is A Data Warehouse?
Comparing Operational Applications And Data Warehouses
Data Warehouse Overview
Development Approach
Data Sources
Staging Tables
Data Warehouse Model
Data Warehouse Design
Data Warehouse Data
End User Access, Old Data, And Metadata Management
Introduction To The Case Study
Data Sources
Data Modeling Review - Part 1
Data Modeling Review - Part 2
Data Sources Overview
Source Data: Menu Definition
Source Data: Miscellaneous Metadata
Source Data: Customer Order
Source Data: Customer Account
Source Data: Customer Prospect
Source Data: Vendor Procurement
Case Study: Assess Source Data
Staging Tables
Staging Tables Overview
Case Study: Create Staging Model
Data Warehouse Modeling Basics
The Star Schema
Dimension
Fact
Surrogate Keys
The Bus Architecture
Dimensional Modeling And Agile Development
Practical Tips
Self Assessment Test
Case Study: Business Requirements
Case Study: Bus Architecture
Recurrent Dimensions
Date
Time
Customer
Account
Employee
Unit of Measure
Product
Currency
Audit
Case Study: Initial Warehouse Model - Part 1
Case Study: Initial Warehouse Model - Part 2
Case Study: Initial Warehouse Model - Part 3
DW Modeling - Advanced Dimension
Kinds of Conformed Dimensions
Junk Dimension
Degenerate Dimension
Slowly Changing Dimension - Part 1
Slowly Changing Dimension - Part 2
Snowflake, Outrigger, and Bridge
Swappable Dimension
Master Dimension
Hierarchy
Practical Tips
Self Assessment Test
Case Study: Elaborate Dimensions
DW Modeling - Advanced Fact
Kinds Of Facts
Transaction Fact
Periodic Snapshot
Accumulating Snapshot
Aggregate Fact
Consolidated Fact
Practical Tips
Case Study: Elaborate Facts
Data Warehouse Modeling Recap
Warehouse Modeling Review
Common Warehouse Modeling Mistakes
Data Warehouse Design
Conceptual, Logical, Physical Models
System Attributes - Part 1
System Attributes - Part 2
Data Types And Domains
Nullability
Constraints
Data Warehouse Tuning - Part 1
Data Warehouse Tuning - Part 2
Views - Part 1
Views - Part 2
Miscellaneous Aspects Of Design
Practical Tips
Self Assessment Test
Case Study: Create Staging SQL
Case Study: Execute Staging SQL
Case Study: Create Warehouse SQL
Case Study: Execute Warehouse SQL
Data Warehouse Data
Warehouse Data Overview
Source-To-Target Mappings
Data Profiling
Loading Staging Tables - Part 1
Loading Staging Tables - Part 2
Loading The Date and Time Dimensions - Part 1
Loading The Date and Time Dimensions - Part 2
Initial Warehouse Loading: Dimensions
Initial Warehouse Loading: Facts
Updating The Warehouse
Warehouse Data Processing And Agile Development
Case Study: Load Warehouse Data
End User Access
End User Access Overview
Case Study: Analyze Data - Part 1
Case Study: Analyze Data - Part 2
Data And Metadata Management
Offload Of Old Data
Metadata Management
Conclusion
Course Wrap-Up
In Search Of Database Nirvana
The Swinging Database Pendulum
Hybrid Transaction/Analytical Processing Workloads
Query Versus Storage Engines
The Challenges Of HTAP
Getting Started
Introduction To Elasticsearch
About The Author
Basic Operations
Installing And Configuring Elasticsearch
Document CRUD - Creating, Retrieving, Updating And Deleting
Running Searches And Aggregations
Data Structure
Mappings And Predefined Fields
Core Types For Your Own Fields
Using Predefined And Custom Analyzers
Queries And Relevance
Returning Specific Fields, Sorting And Pagination
Full-Text Search With Match And Multi-Match Queries
Using The Lucene Query Syntax In Query Strings
Combining Full-Text And Term-Oriented Queries With The Bool Query
Tuning Relevance
Aggregations
Using Queries And Aggregations Together In A Cluster
Combining Different Kinds Of Aggregations
Important Aggregation Types
Document Relationships
Objects And Nested Documents
Parent-Child Relations
Denormalizing And Application-Side Joins
Performance And Scaling
Optimizing Indexing And Searching
Optimizing Node Settings
Configuring Shards And Replicas
Scaling Strategies
Monitoring And Administration
Easy Maintenance With Aliases And Index Templates
Tuning Your Cluster For Stability
Monitoring Elasticsearch Logs And Metrics
Backups And Upgrades
Conclusion
Course Wrap-Up
Data Model And Architecture
Introduction To Accumulo
About The Author
The Accumulo Data Model
Architecture
Working With Accumulo
Installation And Configuration
Running And Monitoring
Using The Shell
Basic Application Development
Starting Development
Writing Data
Reading Data
Table API
Application Security
Authentication
Authorization
Intermediate Application Development
Updates And Deletes
Writing Secondary Indexes
Reading Secondary Indexes
Handling Hardware Failure
Advanced Application Development
Mapreduce
Spark
Iterators
Thrift Proxy
Performance
Table Design
Optimization Features
Administration
Monitoring
Table Management
Importing And Exporting Tables
Cluster Changes
Replication
Conclusion
Conclusion
Course Overview
About This Course
About The Instructor
Course Sittings
At The End Of This Course
Tooling
Initializing Amazon Web Services
Using Cloudera Director To Spin Up A Test Cluster
Crash Course In Cloudera Manager
Hadoop Insecurities
Permissions And Encryption
Where Permissions Stop
Hive: Transform Harmful
Authentication With MIT Kerberos
Installing MIT Kerberos
Enabling Kerberos Authentication
Using MIT Kerberos
Submitting Jobs And Running Queries With Kerberos Auth
Authentication With Active Directory
Installing An AD Server
Preparing AD Server For Hadoop
Impala LDAP Authentication With Active Directory
Using Hue With Active Directory
Preparing Cluster With Kerberos Authentication
Running The CM Wizard
Using Kerberos
Sharing Kerberos Tickets With Active Directory
Authorization
No Authorization
Enabling Sentry Authorization
Using Sentry - Defining Roles
Using Sentry - Querying With Hue
Custom Code And Hive UDFs With Sentry
HDFS Extended ACLs
HDFS Sentry Sync
Sentry Authentication With Solr - Part 1
Sentry Authentication With Solr - Part 2
Encryption
Creating An HDFS Encryption Zone
Using HDFS Encryption Zones
SSL: Crash Course In SSL Tools
SSL: Preparing A Cluster For SSL Using A Self-Signed Root CA
SSL: Enabling SSL For HDFS And Yarn
SSL: Verifying SSL With HDFS And Yarn
SASL Hive And HiveServer2
SSL With HBase And Oozie
SSL With Impala
SSL With Hue
Developer Topics
UserGroupInformation Basics
Delegation Tokens
Secure Impersonation
Administrator Topics
Role Assignments And Gateway Isolation
Hbase ACLs
Audits
Sqoop
Joining An AD Domain
Secure Hadoop Topics
The Secure Hadoop Market
Cheats
Conclusion
Wrap Up