Streaming analytics (or fast data processing) is becoming an increasingly popular subject in enterprise organizations. Customers want real-time experiences, such as notifications and advice based on their online behavior and other users’ actions. A typical streaming analytics solution follows a “pipes and filters” pattern that consists of three main steps: detecting patterns on raw event data (complex event processing), evaluating outcomes with the aid of business rules and machine learning algorithms, and deciding on the next action. At the core of this architecture is the execution of predictive models that operate on enormous amounts of never-ending data streams.

Bas Geerdink (Aizonic) presents an architecture for streaming analytics solutions that covers many use cases that follow this pattern: actionable insights, fraud detection, log parsing, traffic analysis, factory data, the IoT, and others. He explores a few architecture challenges that arise when dealing with streaming data, such as latency issues, event time versus server time, and exactly once processing. The solution is open source and available on GitHub: build on the KISSS stack.

Prerequisite knowledge

• A basic understanding of big data and fast data applications and application and solution architecture
• Experience with a reference architecture

What you'll learn

• Learn to set up a streaming analytics (fast data) solution, basic concepts in this field, and the KISSS stack

This session is from the 2019 O'Reilly Strata Conference in New York, NY.