stream based architecture

@_tommichiels_#Devoxx #Streambased

Stream Based Architecture punching and crunching

[email protected]

mailto:[email protected]?subject=


During this talk I will try to convince you that we need to evolve to stream based architectures


Setting the stageOur actors

Punchers

Crunchers


Birth, First Encounter, The LoveStory


Crunchers life


1950 1970 1990

Machine Code

Imperative

Crunchers life

Declarative


Punchers live


Data models

Flat

Hierarchical

Network

RelationalNavigational


Normalisation

Isolate data so that additions, deletions, and modifications can be made in just

one table and then propagated through the rest of the database using the defined foreign keys


1950 1970 1990

Flat

Navigational

Punchers life

Relational


The marriage, thx Edgar


Normalisation



Wedding vows

SQL +


Crunch(calculate)

Punch(store)

Logic

One happy family


And then it started to get complicate

Perf

orm

ance

Complexity


PerformanceConcurrent users/operations


Performance


Complexity(Re)act to changing world


ComplexityHistorical reasons


ComplexityLegacy


Puncher problems

Perf

orm

ance

• Persistence / IO ?•Model ?


Cruncher problems

Complexity

•Abstractions ?•Reuse?


Crunch(calculate)

Punch(store)

Logic

Scaling up(centralised)


Crunch(calculate)

Punch(store)

Logic

Scaling out(distributed)


Propagating Through

•File•DB•RPC•Message


Centralised vs distributed

Perf

orm

ance

Complexity

Distributed

Centralised


Layers

Crunch(calculate)

Punch(store)

Logic

Punch(store)

UI


2 vs N tier

Perf

orm

ance

Complexity

Distributed

Centralised

2 tier N tier


DataDriven

Perf

orm

ance

Complexity

Distributed

Centralised

DataDriven

2 tier N tier


Query

OLTP

ETLExtract

Transform

Load

OLAP


Abstraction

Reuse

Crunchers midlife crisis


1950 1970 1990

Machine Code

ProceduralObject Oriented

Crunchers midlife crisis


AppDriven

Perf

orm

ance

Complexity

Distributed

Centralised

AppDriven

2 tier N tier


ORM Hell

Impedance Mismatch


Monolith Hell


Spaghetti integration


Service Oriented

Perf

orm

ance

Complexity

Distributed

CentralisedService Oriented

2 tier N tier


Satellite Services

client

Service interface

Data aggregates

O/R mapping

RDBMS


Punchers midlife crisis

©



•Partitions•Eventual Consistence

First Class Citizens


MicroServices

client

NOSql

Service


What about reports?


DataLake


Distributed Data Driven

Perf

orm

ance

Complexity

Distributed

Centralised

2 tier N tier

Distributed DataDriven


Distributed Data Driven

Perf

orm

ance

Complexity

Distributed

Centralised

2 tier N tier

Distributed DataDriven

DataDrivenAppDriven

Service Oriented


How did we end up here!!!!


How did we end up here!!!!

Shared DB

Messaging

RPC

File


The Divorce(ETL)


Remember our wedding vows




Get us out of this mess!


Immutable

an immutable object is an object whose state cannot be modified after it is created

10


Stream

10 10 - 10 10 8 vs

is a (ordered) sequence of immutable objects


Stream Computing

10 + 10+ (- 10) = 10

data

state is derivedintent


Back to Maths

•Associativity•Commutative


Back to Maths

10 + 10+ (- 10) = 10

Type

Binary Operation


Back to Maths

A ∪ B

Type

Binary Operation


Birth of a commutative monoid

(10 + 10)+ (- 10) = ((-10) +10) + 10

Calculation can be partitioned

How we combine intermediate results is not important


Programming with solid base in maths


Examples of streams

•Logfiles•Journals/Ledgers•Atom Feeds


Propagating New Style

•Make stream available•Consumers keep track of consumed events

•Producers Schema on Write


Wedding vows renewed

Propagate immutable events by using streams and use stream crunching as local as possible.


Wedding vows renewed

Not Only ETL


Streams on Micro(Service) Level

•EventSourcing•CQRS


Streams on Micro Level


Streams on Macro level



kappa architecture


Reports vs Dashboards


Fast Data


• BloomFilter

• HyperLogLog

• Count-MinSketch

• MinHash

Stream processing


Smack Stack https://github.com/killrweather/killrweather


Happily Ever After

stream based architecture

Technology