oak , the architecture of the new repository

APACHE SLING & FRIENDS TECH MEETUPBERLIN, 22-24 SEPTEMBER 2014

Oak, the Architecture of the new RepositoryMichael Dürig, Adobe Research

Design goals

adaptTo() 2014 2

Scalable Big repositories Clustering Customisable, flexible OSGi friendly

Outline

adaptTo() 2014 3

CRUD Changes Search

Tree model

adaptTo() 2014 4

a d

b c

Updating

adaptTo() 2014 5

?

x

a d

b c

MVCC

adaptTo() 2014 6

a d

b c

r2: /dr1: /d

r1: /a/br2: /a/b

r1: / r2: /

r2: /d/x

HEAD

adaptTo() 2014 7

Refresh and Garbage Collection

garbage

Refresh

adaptTo() 2014 8

garbage

Garbage collection

adaptTo() 2014 9

adaptTo() 2014 10

Concurrency and Conflicts

r2a

Concurrent updates

adaptTo() 2014 11

r2br1

Merging

adaptTo() 2014 12

r1 r3merge

r2b

r2a

upates

Conflict handling: serialisation

adaptTo() 2014 13

Fully serialised Fail, no concurrent update

Partially serialised Concurrent conflict free updates

Conflict handling strategies: merging

adaptTo() 2014 14

Partial merge Conflict markers, deferred resolution

Full merge Need to choose victim

adaptTo() 2014 15

Replicas and Sharding

Replica and caches

adaptTo() 2014 16

master copy full replica cache

Sharding strategies

adaptTo() 2014 17

by path by level by hash with caching

adaptTo() 2014 18

Implementations

MicroKernel / NodeStore

adaptTo() 2014 19

Tree / Revision model implementation

Responsible for

Clustering

Sharding

Caching

Conflict handling

Not responsible for

Validation

Access control

Search

Versioning

Current implementations

adaptTo() 2014 20

DocumentMK TarMK (SegmentMK)

Persistence MongoDB, JDBC Local FS

Conflict handling Partial serialisation Full serialisation

Clustering MongoDB clustering Simple failover

Sharding MongoDB sharding N/A

Node Performance Moderate High

Key use cases Large deployments (>1TB), concurrent writes

Small/medium deployments, mostly read

adaptTo() 2014 21

Access Control

Accessible paths

adaptTo() 2014 22

a d

cb

xistentialism

adaptTo() 2014 23

All paths traversable Node may not exist Decorator on NodeStore

E

root.getChildNode("a").exists();

root.getChildNode("a") .getChildNode("b").exists();

⟹ false

⟹ true

adaptTo() 2014 24

Comparing Revisions

Content diff

adaptTo() 2014 25

What changed between trees Cornerstone for

Validation Indexing Observation …

What changed?

adaptTo() 2014 26

∆

Example: merging

adaptTo() 2014 27

r1

r2a

r2b

r3

r1 ➞ r2a“a” modified

“b” removed

∆

r1 ➞ r2b “d” modified

“x” added

∆

adaptTo() 2014 28

Commit Hooks

Commit hooks

adaptTo() 2014 29

Key plugin mechanism Higher level functionality

Validation (node type, access control, …) Trigger (auto create, defaults, …) Updates (index, …)

Editing a commit

adaptTo() 2014 30

∆ ∆ + x

Commit hooks

adaptTo() 2014 31

Based on content diff pass a commit fail a commit edit a commit

Applied in sequence

Type of hooks

adaptTo() 2014 32

CommitHook Editor Validator

Content diff Optional Always Always

Can modify Yes Yes No

Programming model

Simple Callbacks Callbacks

Performance impact

High Medium Low

adaptTo() 2014 33

Observers

Observers

adaptTo() 2014 34

Observe changes After commit Often does a content diff Asynchronous Optionally synchronous

Local cluster node only

Examples

adaptTo() 2014 35

JCR observation External index update Cache invalidation Logging

adaptTo() 2014 36

Search

Query Engine

adaptTo() 2014 37

SELECTWHERE x=y

/a//*

Parser

Parser

Parser

Index

Index

Traverse

Parser

Index

parse execute post process

Index Implementations

adaptTo() 2014 38

Property (ordered) Reference Lucene

In-content or file system Solr

Embedded or external

adaptTo() 2014 39

Big Picture

Big picture

adaptTo() 2014 40

MicroKernel

Oak Core

Oak JCR

Oak API

NodeStore API

JCR API

Plugins

Resources

adaptTo() 2014 41

http://jackrabbit.apache.org/oak/

adaptTo() 2014 42

Appendix

Resources

adaptTo() 2014 43

http://jackrabbit.apache.org/oak/

http://jackrabbit.apache.org/oak/docs/

https://svn.apache.org/repos/asf/jackrabbit/oak/trunk

/