apache jackrabbit oak on mongodb

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Marcel Reutegger | Senior Software Engineer Apache Jackrabbit Oak on MongoDB

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak

2

About me

Software Engineer

At Day/Adobe since 2002

JCR API Specification

Apache member

Apache Jackrabbit / Oak

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Contents

Adobe Experience Manager

Java Content Repository

Apache Jackrabbit Oak

Multiversion Concurrency Control

Transactions

Content Addressable Storage

Q & A

3

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Adobe Experience Manager

4

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Adobe Experience Manager – Technology Stack

5

OSGi Container

Web Framework

Java Content Repository

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Adobe Experience Manager – Technology Stack

6

OSGi Container

Web Framework

Java Content Repository

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Adobe Experience Manager – Technology Stack

7

Java Content Repository / Apache Jackrabbit Oak

MongoDB RDBMS Tar

.tar

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Contents

Adobe Experience Manager

Java Content Repository

Apache Jackrabbit Oak

Multiversion Concurrency Control

Transactions

Content Addressable Storage

Q & A

8

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Java Content Repository – Why?

9

Developer happiness

“Build me a web content management system!”

Easy: LAMP stack

Linux Apache

MySQL PHP

Done in 2 weeks

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Java Content Repository – Why?

10

Developer happiness

“Nice, but I want to organize my pages in a hierarchy.”

Apply a well known hierarchical database model and update the application.

Done in 4 weeks

Build me a web content management system!

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Java Content Repository – Why?

11

“Can you please add structured and fulltext searches?”

Integrate with Apache Solr or Elasticsearch

Done in 4 weeks

Nice, but I want to organize my pages in a hierarchy.

Developer happiness

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Java Content Repository – Why?

12

“I accidentally deleted the product page. We need to version our content.”

Introduce new tables and rewrite the application.

Done in 8 weeks

Can you please add structured and fulltext searches?

Developer happiness

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Java Content Repository – Why?

13

“We cannot publish financial results, unless the system has fine grained access control.”

Introduce more tables and integrate with a directory server.

I’ll get back to you next year!

“I accidentally deleted the product page. We need to version our content.”

Developer happiness

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Java Content Repository – Features

JSR-283 – JCR 2.0 released 2009

Hierarchical - Structured and binary data

Query – SQL, XPath and Java language binding

Access Control on Node and Property level

Versioning – Modeled after WebDAV DeltaV (RFC 3253)

Locking – Shallow or deep

Asynchronous Observation

14

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Contents

Adobe Experience Manager

Java Content Repository

Apache Jackrabbit Oak

Multiversion Concurrency Control

Transactions

Content Addressable Storage

Q & A

15

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – Design

16

Cacheable

Customizable

Scalable Support NoSQL Storage Support Sharding

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – Design

17

Cacheable

Customizable

Scalable

Pluggable Storage Custom Index Definitions

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – Design

18

Cacheable

Customizable

Scalable

Multiversion Concurrency Copy-On-Write

Content Addressable Storage

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Contents

Adobe Experience Manager

Java Content Repository

Apache Jackrabbit Oak

Multiversion Concurrency Control

Transactions

Content Addressable Storage

Q & A

19

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – MVCC & Copy-On-Write

20

Rev 1

/

/a /b

/a/1 /a/2 /b/1

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – MVCC & Copy-On-Write

21

/a /b

/a/1 /a/2 /b/1 /b/2

Rev 1

/

new

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – MVCC & Copy-On-Write

22

/a /b

/a/1 /a/2 /b/1 /b/2

Rev 1

/b'

/ /'

Rev 2 copy parents

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – MVCC & Copy-On-Write

23

/a /b

/a/1 /a/2 /b/1 /b/2

Rev 1

/b'

/ /'

Rev 2

concurrent access to Rev 1 and Rev 2

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – MVCC & Copy-On-Write

24

/a

/a/1 /a/2 /b/1 /b/2

/b'

/'

Rev 2 collect garbage

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – MVCC & Copy-On-Write

25

/a /b'

/a/1 /a/2 /b/1 /b/2

Rev 2

/'

compact

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – MVCC & Copy-On-Write

26

⊕

⊖

Stable snapshot view of data Writes do not block reads

Higher storage cost Garbage collection

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – The Data Model

27

{! _id : “/home/john”,! name : “john”,! email : “john@example.com”!}!

{! _id : “2:/home/john”,! name : { “r14979e4b424-0-1” : “john” },! email : { “r14979e4b424-0-1” : “john@example.com” },! _deleted : { “r14979e4b424-0-1” : “false” },! _revisions : { “r14979e4b424-0-1” : “c” }!}!

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – The Data Model

28

{! _id : “2:/home/john”,! name : { “r14979e4b424-0-1” : “john” },! email : { “r14979e4b424-0-1” : “john@example.com” },! _deleted : { “r14979e4b424-0-1” : “false” },! _revisions : { “r14979e4b424-0-1” : “c” }!}!

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – The Data Model

29

{! _id : “2:/home/john”,! name : { “r14979e4b424-0-1” : “john” },! email : { “r14979e4b424-0-1” : “john@example.com” },! _deleted : { “r14979e4b424-0-1” : “false” },! _revisions : { “r14979e4b424-0-1” : “c” }!}!

Timestamp Counter Cluster ID

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – The Data Model

30

{! _id : “2:/home/john”,! name : { “r14979e4b424-0-1” : “john” },! email : {! “r14979e4b424-0-1” : “john@example.com”,! “r14979e6a941-0-1” : “john.doe@example.com”! },! _deleted : { “r14979e4b424-0-1” : “false” },! _revisions : {! “r14979e4b424-0-1” : “c”,! “r14979e6a941-0-1” : “c”! }!}!

Change email to “john.doe@example.com”

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Contents

Adobe Experience Manager

Java Content Repository

Apache Jackrabbit Oak

Multiversion Concurrency Control

Transactions

Content Addressable Storage

Q & A

31

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – Transactions

32

{! _id : “2:/home/john”,! name : { “r14979e4b424-0-1” : “john” },! _deleted : { “r14979e4b424-0-1” : “false” },! _commitRoot : { “r14979e4b424-0-1” : “1” }!}!

{! _id : “3:/home/john/profile”,! avatar : { “r14979e4b424-0-1” : <bin> },! _deleted : { “r14979e4b424-0-1” : “false” },! _commitRoot : { “r14979e4b424-0-1” : “1” }!}!

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – Transactions

33

{! _id : “2:/home/john”,! name : { “r14979e4b424-0-1” : “john” },! _deleted : { “r14979e4b424-0-1” : “false” },! _commitRoot : { “r14979e4b424-0-1” : “1” }!}!

{! _id : “3:/home/john/profile”,! avatar : { “r14979e4b424-0-1” : <bin> },! _deleted : { “r14979e4b424-0-1” : “false” },! _commitRoot : { “r14979e4b424-0-1” : “1” }!}!

{! _id : “1:/home”,! _deleted : { “r14979e1b312-0-1” : “false” },! _revisions : {! “r14979e1b312-0-1” : “c”,! “r14979e4b424-0-1” : “c”! }!}!

Conditional update for commit:

{! _id : “1:/home”, ! “_collisions.r14979e4b424-0-1” : { $exists : false }!}!

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – Transactions

34

{! _id : “3:/home/john/profile”,! avatar : {! “r14979e4b424-0-1” : <bin>! },! _deleted : {! “r14979e4b424-0-1” : “false”! },! _commitRoot : {! “r14979e4b424-0-1” : “1” ! }!}!

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – Transactions

35

{! _id : “3:/home/john/profile”,! avatar : {! “r14979e4b424-0-1” : <bin>,! “r14979e6c7a2-0-1” : <bin>! },! _deleted : {! “r14979e4b424-0-1” : “false”! },! _commitRoot : {! “r14979e4b424-0-1” : “1”,! “r14979e6c7a2-0-1” : “1” ! }!}!

T-1

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – Transactions

36

{! _id : “3:/home/john/profile”,! avatar : {! “r14979e4b424-0-1” : <bin>,! “r14979e6c7a2-0-1” : <bin>,! “r14979e6c7a3-0-1” : <bin>! },! _deleted : {! “r14979e4b424-0-1” : “false”! },! _commitRoot : {! “r14979e4b424-0-1” : “1”,! “r14979e6c7a2-0-1” : “1”,! “r14979e6c7a3-0-1” : “1”! }!}!

T-1 T-2

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – Transactions

37

{! _id : “1:/home”,! _deleted : { ! “r14979e1b312-0-1” : “false”! },! _revisions : {! “r14979e1b312-0-1” : “c”! },! _collisions : {! “r14979e6c7a2-0-1” : “true”! }!}!

T-2

Conditional update for collision marker: {! _id : “1:/home”, ! “_revisions.r14979e6c7a2-0-1” : { $exists : false }!}!

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – Transactions

38

{! _id : “1:/home”,! _deleted : { ! “r14979e1b312-0-1” : “false”! },! _revisions : {! “r14979e1b312-0-1” : “c”! },! _collisions : {! “r14979e6c7a2-0-1” : “true”! }!}!

T-2

Conditional update for commit: {! _id : “1:/home”, ! “_collisions.r14979e6c7a2-0-1” : { $exists : false }!}!

T-1

✗

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – Transactions

39

{! _id : “1:/home”,! _deleted : { ! “r14979e1b312-0-1” : “false”! },! _revisions : {! “r14979e1b312-0-1” : “c”,! “r14979e6c7a3-0-1” : “c”! },! _collisions : {! “r14979e6c7a2-0-1” : “true”! }!}!

T-2 T-1

✗

{! _id : “1:/home”, ! “_collisions.r14979e6c7a3-0-1” : { $exists : false }!}!

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – Transactions

40

{! _id : “3:/home/john/profile”,! avatar : {! “r14979e4b424-0-1” : <bin>,! “r14979e6c7a2-0-1” : <bin>,! “r14979e6c7a3-0-1” : <bin>! },! _deleted : {! “r14979e4b424-0-1” : “false”! },! _commitRoot : {! “r14979e4b424-0-1” : “1”,! “r14979e6c7a2-0-1” : “1”,! “r14979e6c7a3-0-1” : “1”! }!}!

T-1

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Contents

Adobe Experience Manager

Java Content Repository

Apache Jackrabbit Oak

Multiversion Concurrency Control

Transactions

Content Addressable Storage

Q & A

41

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – Content Addressable Storage

42

GridFS

file chunk 0

chunk 1

chunk N

…

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – Content Addressable Storage

43

Oak

Binary Storage Node Storage

Binary

Chunk 0x38a7

Chunk 0xc92a

Chunk 0x8f91

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – Content Addressable Storage

44

Oak

Binary Storage

Binary 0x38a7

0x38a7

Chunk 0x38a7

Chunk 0xc92a

Chunk 0x8f91

Hash:

Node Storage

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Binary

Apache Jackrabbit Oak – Content Addressable Storage

45

Oak

Binary Storage

0x8f91

Chunk 0x38a7

Chunk 0xc92a

Chunk 0x8f91

Hash:

0x38a7 0x8f91

Node Storage

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Binary

Apache Jackrabbit Oak – Content Addressable Storage

46

Oak

Binary Storage

0x52f1

Chunk 0x38a7

Chunk 0xc92a

Chunk 0x8f91

Hash:

0x38a7 0x8f91

Chunk 0x52f1

0x52f1

Node Storage

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – Content Addressable Storage

47

Oak

Binary Storage

Chunk 0x38a7

Chunk 0xc92a

Chunk 0x8f91

Chunk 0x52f1

Node Storage

0x38a7 0x8f91 0x52f1

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Apache Jackrabbit Oak – Content Addressable Storage

48

⊕

⊖

De-duplication on chunk level Chunks are immutable Shared Storage

Garbage collection

© 2014 Adobe Systems Incorporated. All Rights Reserved.

Summary

Adobe Experience Manager

Java Content Repository

Apache Jackrabbit Oak

Multiversion Concurrency Control

Transactions

Content Addressable Storage

Q & A

49

© 2014 Adobe Systems Incorporated. All Rights Reserved. 50

Q & A

© 2014 Adobe Systems Incorporated. All Rights Reserved.