append only data stores

vs

Append-only data stores

onsdag 16 oktober 13

What?


Append-only data stores

Only add, never remove or change

Can retrieve old values


Source code

Version controlled

Keep all versions


The problem


Mutable state

id name email

jakr Jan Kronquist [email protected]


Mutable state

id name email


UPDATE customerSET email="[email protected]"WHERE id="jakr"


Mutable state

id name email



I didn’t get confirmation of the order last week?


Mutable state

id name email



I didn’t get confirmation of the order last week?

Could you reset my email address?


The technical details

SELECT count(*) AS inSwedenFROM customerWHERE email LIKE "%@jayway.se"

SELECT count(*) AS elsewhereFROM customerWHERE email LIKE "%@jayway.com"

total = inSweden + elsewhere


Human errors

DELETEFROM customerWHERE email LIKE "j%"


Scalability - the easy parts

Static content

Caching



Static content

Stateless work

Caching



Static content

Stateless work

Load balancer

Caching


State + behavior

Scalability - the hard parts


Example domain


Rock - Paper - Scissors


http://rock-paper-scissors.com/

The future Facebook of Rock Paper Scissors

Millions of users

Many games per user




Playing the game

Player A

Player B

Server


Playing the game

Player A

Player B

Server

Game 123


Playing the game

Player A

Player B

Serverpaper

Game 123


Playing the game

Player A

Player B

Serverpaper

Player B: paper

Game 123


Playing the game

Player A

Player B

Server

rock

Player B: paper

Player A: rock

Game 123


Playing the game

Player A

Player B

Server

Player B: paper

Player A: rock

Game 123

Game 123winner: Player Bloser: Player A


Playing the game

Player A

Player B

Server

Player B: paper

Player A: rock

Game 123

Game 123winner: Player Bloser: Player A

CREATED WAITING

GAME WON

GAME TIED

any move

other move(victory)

other move(tie)


Entities


Entities

Player+ id+ name+ email


Entities

Game+ id+ state+ players+ moves+ winner+ loser



Entities

Game+ id+ state+ players+ moves+ winner+ loser Move

+ move+ player



Actions

Create game

Make move

Get game result

POST /games

POST /games/123 move=rock

GET /games/123


Actions

Create game

Make move

Get game result

POST /games

POST /games/123 move=rock

GET /games/123

1. Form (make move)

2. Waiting

3. Game result

4. Error


Clojure


Clojure data structures



Lists - singly linked(1 2 3 4 5) (fred ethel lucy) (list 1 2 3)




Vectors - indexed access[1 2 3] ["fred" 17 3.14 "bar"]




Vectors - indexed access[1 2 3] ["fred" 17 3.14 "bar"]

Maps{:name "Jan Kronquist" :age 37}


Event Sourcingwith

Event Store


Event Sourcing

Events that have happened

Ordered in time

Source of truth


http://geteventstore.com

Runs on .NET and Mono

Free (store) or Commercial (HA)

APICustom TCP (c#)

ATOM over HTTP (xml/json)

FeaturesEvent streams, projections, generate new events (CEP)

€1500/year




Event example

Server EventStore


Event example

Server

CreateGameCommand{:game 123 :creator "player-1"}

EventStore


Event example

Server


GameCreatedEvent{:game 123 :creator "player-1"}

EventStore


Event example

Server

MakeMoveCommand{:game 123 :player "player-1" :move "rock"}


MoveMadeEvent{:game 123 :player "player-1" :move "rock"}

EventStore


Event example

Server

MakeMoveCommand{:game 123 :player "player-2" :move "paper"}


MoveMadeEvent{:game 123 :player "player-1" :move "rock"}

MoveMadeEvent{:game 123 :player "player-2" :move "paper"}GameWonEvent{:game 123 :winner "player-2" :loser "player-1"}

EventStore


GameCreatedEvent{:game 123, :creator "player-1", :time 1} MoveMadeEvent{:game 123, :player "player-1", :move "rock", :time 2}

Current state?


CREATED WAITING

GAME WON

GAME TIED

any move

if victory

other move

if tie


Current state?


CREATED WAITING

GAME WON

GAME TIED

any move

if victory

other move

if tie

GameCreatedEvent{:game 123, :creator "player-1", :time 1} MoveMadeEvent{:game 123, :player "player-1", :move "rock", :time 2}GameCreatedEvent

Current state?

CREATED


CREATED WAITING

GAME WON

GAME TIED

any move

if victory

other move

if tie


Current state?

WAITING

MoveMadeEvent


Aggregates

Scope of consistency when handling a command


Move+ id+ move+ player



Aggregates as Event Streams

event1 GameCreatedEvent{:game 1...}event2 MoveMadeEvent{:game 1....}

stream = game-1version = 2





event1 GameCreatedEvent{:game 2...}event2 MoveMadeEvent{:game 2....}event3 MoveMadeEvent{:game 2....}event4 GameWonEvent{:game 2....}






event1 GameCreatedEvent{:game 2...}event2 MoveMadeEvent{:game 2....}event3 MoveMadeEvent{:game 2....}event4 GameWonEvent{:game 2....}


curl "http://127.0.0.1:2113/streams/game-1" -d @eventdata.json -H "Content-Type:application/json" -H "ES-ExpectedVersion: 2"


http://127.0.0.1:2113/streams/game-1

http://127.0.0.1:2113/streams/game-1

Projections

GameCreatedEvent{:game 1...}MoveMadeEvent{:game 1....}GameCreatedEvent{:game 2...}MoveMadeEvent{:game 2....}MoveMadeEvent{:game 2....}GameWonEvent{:game 2....}


Projections


for (Event events : allEvents) { if (event instanceof GameCreatedEvent) { created++; }}return created;


Projections



Projections

GameCreatedEvent{:game 1...}MoveMadeEvent{:game 1....}GameCreatedEvent{:game 2...}MoveMadeEvent{:game 2....}MoveMadeEvent{:game 2....}GameWonEvent{:game 2....} created = 2


Projections

GameCreatedEvent{:game 1...}MoveMadeEvent{:game 1....}GameCreatedEvent{:game 2...}MoveMadeEvent{:game 2....}MoveMadeEvent{:game 2....}GameWonEvent{:game 2....} created = 2GameCreatedEvent{:game 3...}


Projections


(state, event) -> { if (event instanceof GameCreatedEvent) { return state++; } rerturn state;}

created = 2GameCreatedEvent{:game 3...}


Projections in JavascriptfromAll().when({ $init: function () { return { created: 0, completed:0 }; },

});



});

GameCreatedEvent: function(s, e) { s.created++; return s; },



});


GameWonEvent: function(s, e) { s.completed++; return s; },



});



GameTiedEvent: function(s, e) { s.completed++; return s; }



});



GameTiedEvent: function(s, e) { s.completed++; return s; }

/projections/games-counter{created: 15,completed: 10

}


External Projections

events

Business logic



events events

Business logic Projections



Consistent

Available

events events

Business logic Projections



Commands Queries

Consistent

Available

events events


System with EventStore



Command Handler



1.command Command Handler



1.command Event Store

Command Handler

2.get events




Command Handler

2.get events

3.add events




Command Handler

2.get events

3.add events

4.updateprojections




Command Handler

2.get events

3.add events

4.updateprojections

5.get eventsor projections

Some service




Command Handler

2.get events

6.update state

3.add events

4.updateprojections


Some service




Command Handler

2.get events

6.update state

3.add events

4.updateprojections

7.query


Some service


More info

Functional Programming with DDDhttp://skillsmatter.com/podcast/design-architecture/ddd-functional-programming

A deep look into the Event Storehttp://vimeo.com/53153270


http://skillsmatter.com/podcast/design-architecture/ddd-functional-programming

http://skillsmatter.com/podcast/design-architecture/ddd-functional-programming

Datomic


http://www.datomic.com/

Runs on JVM

Free (3 peers, local storage) or Commercial (HA, distrib)

FeaturesACID transactions, declarative query engine

€2300




Fact

Something known to have happened or existed


Fact


2004-10-01 the email of Jan was ”[email protected]”


mailto:[email protected]


Fact









Atomic fact (datom)

Entity

Attribute

Value

Time


Atomic fact (datom)

Entity

Attribute

Value

Time

The person Jan Kronquist

email

”[email protected]”

2004-10-01


Data perception

Facts

Add facts

Add more facts

Even more facts

time


Data perception

Facts

Add facts

Add more facts

Even more facts

time

query


Data perception

Facts

Add facts

Add more facts

Even more facts

time

query

query


Data perception

Facts

Add facts

Add more facts

Even more facts

time

query


Database deconstructed



TransactorConsistency by serializing transactions




PeerYour code + Query engine




PeerYour code + Query engine

Storage serviceKey/value store

Scalable reads and writes

eg DynamoDB, InfiniSpan, Riak + ZooKeeper, SQL(!)


System with Datomic

Transactor

Distributed Storage Service

Peer A Peer A


System with Datomic

Transactor


Peer A Peer A

1.command


System with Datomic

Transactor


Peer A Peer A

2.read and cache

1.command


System with Datomic

Transactor


Peer A

3.transact

Peer A

2.read and cache

1.command


System with Datomic

Transactor


Peer A

3.transact

Peer A

4.write

2.read and cache

1.command


System with Datomic

Transactor


Peer A

3.transact

Peer A

4.write

2.read and cache

Peer B

5.notify

1.command


Datomic example

Peer Transactor


Datomic example

Peer


Transactor


Datomic example

Peer


[123 :creator "player-1" T1][123 :state :created T1]

Transactor


Datomic example

Peer



Transactor

["player-1" :email "[email protected]" T0]


Datomic example

Peer



Transactor


Datomic example

Peer

MakeMoveCommand{:game 123 :player "player-1" :move :rock}

[123 :creator "player-1" T1][123 :state :created T1][123_1 :move :rock T2][123_1 :player "player-1" T2][123 :moves 123_1 T2][123 :state :waiting T2]

Transactor


Datomic example

Peer

MakeMoveCommand{:game 123 :player "player-2" :move :paper}

[123 :creator "player-1" T1][123 :state :created T1][123_1 :move :rock T2][123_1 :player "player-1" T2][123 :moves 123_1 T2][123 :state :waiting T2][123_2 :move :paper T3][123_2 :player "player-2" T3][123 :moves 123_2 T3][123 :state :won T3][123 :winner "player-2" T3][123 :loser "player-1" T3]

Transactor


Aggregates?

Only by convention

cas (compare and set)

isComponent on relation


Move+ id+ move+ player


What to find

Where clauses [entity attribute value]

Implicit joins

Call arbitrary function

Query language


Query example - basic

[:find ?email :where [_ :email ?email]]


Query example - basic

[:find ?email :where [_ :email ?email]]

[["[email protected]] ["[email protected]"] ["[email protected]"]]


Query example - aggregation

[:find ?state (?count game) :where [?game :state ?state]]




[[:created 4] [:waiting 1] [:won 7] [:tied 3]]





inProgess = created + waiting





inProgess = created + waiting

fromAll().when({ $init: function () { return { created: 0, completed:0 }; },



GameTiedEvent: function(s, e) { s.completed++; return s; }});

/projections/games-counter{

created: 15,completed: 10

}


Query example - join

[:find ?email (count ?game) :where [?player :email ?email] [?game :winner ?player]]


Query at any time

(q ’[:find ?email :where [_ :email ?email]] db)


Query at any time

(q ’[:find ?email :where [_ :email ?email]] db)

(q ’[:find ?email :where [_ :email ?email]] (as-of db #inst "2013-09-02"))


Query over time

(q ’[:find ?e (count ?tx) :where [?e :email _ ?tx]] db)


Query over time

(q ’[:find ?e (count ?tx) :where [?e :email _ ?tx]] db)

(q ’[:find ?e (count ?tx) :where [?e :email _ ?tx]] (history db))


Java API

List<Object> results = Peer.q(someQuery, conn.db());

Future<Map> txResult = conn.transact(data_tx);


More info

http://www.datomic.com/videos.html




Comparison


Comparison

Data model

Schema evolution

Queries

Recommendation


EventStore Events

✓ Understandable by normal people

✓ Designed for integration

✓ Forces aggregate design (scalability)

✓ Capture intent

๏ Require design effort to get right

๏ Complect transaction and query


Datomic Facts

✓ Support partial information

✓ Normalization support

✓ CRUD out of the box

๏ Static schema


✓ New events

✓ New event attributes

✓ Event attributes rename (change deserialization)

✓ Projected state change (rebuild projection)

๏ Cross event refactoring

EventStore schema evolution


✓ New or removed attributes

✓ New or removed entity types

✓Move attributes between entities

๏ Fact attributes rename not possible

๏ Fact attribute type change not possible

Datomic schema evolution


✓ Projections use JavaScript

✓ Persistent

๏ Require projection

EventStore queries


✓ Declarative

✓ Logic-based

✓ Allows calling out to your own code

๏ New language to learn

Datomic queries


When to use Event Sourcing?

Domains where events seem natural

Different teams with different models


When to use Datomic?

Data can be modelled using entities and attributes

History is or may be interesting


More details

http://www.jayway.com/author/jankronquist/

https://github.com/jankronquist

http://www.slideshare.net/jankronquist








Questions?


append only data stores

Entertainment & Humor