apache jackrabbit oak on mongodb
TRANSCRIPT
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Marcel Reutegger | Senior Software Engineer Apache Jackrabbit Oak on MongoDB
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak
2
About me
Software Engineer
At Day/Adobe since 2002
JCR API Specification
Apache member
Apache Jackrabbit / Oak
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Contents
Adobe Experience Manager
Java Content Repository
Apache Jackrabbit Oak
Multiversion Concurrency Control
Transactions
Content Addressable Storage
Q & A
3
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Adobe Experience Manager
4
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Adobe Experience Manager – Technology Stack
5
OSGi Container
Web Framework
Java Content Repository
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Adobe Experience Manager – Technology Stack
6
OSGi Container
Web Framework
Java Content Repository
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Adobe Experience Manager – Technology Stack
7
Java Content Repository / Apache Jackrabbit Oak
MongoDB RDBMS Tar
.tar
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Contents
Adobe Experience Manager
Java Content Repository
Apache Jackrabbit Oak
Multiversion Concurrency Control
Transactions
Content Addressable Storage
Q & A
8
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Java Content Repository – Why?
9
Developer happiness
“Build me a web content management system!”
Easy: LAMP stack
Linux Apache
MySQL PHP
Done in 2 weeks
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Java Content Repository – Why?
10
Developer happiness
“Nice, but I want to organize my pages in a hierarchy.”
Apply a well known hierarchical database model and update the application.
Done in 4 weeks
Build me a web content management system!
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Java Content Repository – Why?
11
“Can you please add structured and fulltext searches?”
Integrate with Apache Solr or Elasticsearch
Done in 4 weeks
Nice, but I want to organize my pages in a hierarchy.
Developer happiness
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Java Content Repository – Why?
12
“I accidentally deleted the product page. We need to version our content.”
Introduce new tables and rewrite the application.
Done in 8 weeks
Can you please add structured and fulltext searches?
Developer happiness
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Java Content Repository – Why?
13
“We cannot publish financial results, unless the system has fine grained access control.”
Introduce more tables and integrate with a directory server.
I’ll get back to you next year!
“I accidentally deleted the product page. We need to version our content.”
Developer happiness
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Java Content Repository – Features
JSR-283 – JCR 2.0 released 2009
Hierarchical - Structured and binary data
Query – SQL, XPath and Java language binding
Access Control on Node and Property level
Versioning – Modeled after WebDAV DeltaV (RFC 3253)
Locking – Shallow or deep
Asynchronous Observation
14
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Contents
Adobe Experience Manager
Java Content Repository
Apache Jackrabbit Oak
Multiversion Concurrency Control
Transactions
Content Addressable Storage
Q & A
15
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – Design
16
Cacheable
Customizable
Scalable Support NoSQL Storage Support Sharding
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – Design
17
Cacheable
Customizable
Scalable
Pluggable Storage Custom Index Definitions
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – Design
18
Cacheable
Customizable
Scalable
Multiversion Concurrency Copy-On-Write
Content Addressable Storage
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Contents
Adobe Experience Manager
Java Content Repository
Apache Jackrabbit Oak
Multiversion Concurrency Control
Transactions
Content Addressable Storage
Q & A
19
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – MVCC & Copy-On-Write
20
Rev 1
/
/a /b
/a/1 /a/2 /b/1
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – MVCC & Copy-On-Write
21
/a /b
/a/1 /a/2 /b/1 /b/2
Rev 1
/
new
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – MVCC & Copy-On-Write
22
/a /b
/a/1 /a/2 /b/1 /b/2
Rev 1
/b'
/ /'
Rev 2 copy parents
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – MVCC & Copy-On-Write
23
/a /b
/a/1 /a/2 /b/1 /b/2
Rev 1
/b'
/ /'
Rev 2
concurrent access to Rev 1 and Rev 2
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – MVCC & Copy-On-Write
24
/a
/a/1 /a/2 /b/1 /b/2
/b'
/'
Rev 2 collect garbage
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – MVCC & Copy-On-Write
25
/a /b'
/a/1 /a/2 /b/1 /b/2
Rev 2
/'
compact
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – MVCC & Copy-On-Write
26
⊕
⊖
Stable snapshot view of data Writes do not block reads
Higher storage cost Garbage collection
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – The Data Model
27
{! _id : “/home/john”,! name : “john”,! email : “[email protected]”!}!
{! _id : “2:/home/john”,! name : { “r14979e4b424-0-1” : “john” },! email : { “r14979e4b424-0-1” : “[email protected]” },! _deleted : { “r14979e4b424-0-1” : “false” },! _revisions : { “r14979e4b424-0-1” : “c” }!}!
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – The Data Model
28
{! _id : “2:/home/john”,! name : { “r14979e4b424-0-1” : “john” },! email : { “r14979e4b424-0-1” : “[email protected]” },! _deleted : { “r14979e4b424-0-1” : “false” },! _revisions : { “r14979e4b424-0-1” : “c” }!}!
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – The Data Model
29
{! _id : “2:/home/john”,! name : { “r14979e4b424-0-1” : “john” },! email : { “r14979e4b424-0-1” : “[email protected]” },! _deleted : { “r14979e4b424-0-1” : “false” },! _revisions : { “r14979e4b424-0-1” : “c” }!}!
Timestamp Counter Cluster ID
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – The Data Model
30
{! _id : “2:/home/john”,! name : { “r14979e4b424-0-1” : “john” },! email : {! “r14979e4b424-0-1” : “[email protected]”,! “r14979e6a941-0-1” : “[email protected]”! },! _deleted : { “r14979e4b424-0-1” : “false” },! _revisions : {! “r14979e4b424-0-1” : “c”,! “r14979e6a941-0-1” : “c”! }!}!
Change email to “[email protected]”
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Contents
Adobe Experience Manager
Java Content Repository
Apache Jackrabbit Oak
Multiversion Concurrency Control
Transactions
Content Addressable Storage
Q & A
31
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – Transactions
32
{! _id : “2:/home/john”,! name : { “r14979e4b424-0-1” : “john” },! _deleted : { “r14979e4b424-0-1” : “false” },! _commitRoot : { “r14979e4b424-0-1” : “1” }!}!
{! _id : “3:/home/john/profile”,! avatar : { “r14979e4b424-0-1” : <bin> },! _deleted : { “r14979e4b424-0-1” : “false” },! _commitRoot : { “r14979e4b424-0-1” : “1” }!}!
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – Transactions
33
{! _id : “2:/home/john”,! name : { “r14979e4b424-0-1” : “john” },! _deleted : { “r14979e4b424-0-1” : “false” },! _commitRoot : { “r14979e4b424-0-1” : “1” }!}!
{! _id : “3:/home/john/profile”,! avatar : { “r14979e4b424-0-1” : <bin> },! _deleted : { “r14979e4b424-0-1” : “false” },! _commitRoot : { “r14979e4b424-0-1” : “1” }!}!
{! _id : “1:/home”,! _deleted : { “r14979e1b312-0-1” : “false” },! _revisions : {! “r14979e1b312-0-1” : “c”,! “r14979e4b424-0-1” : “c”! }!}!
Conditional update for commit:
{! _id : “1:/home”, ! “_collisions.r14979e4b424-0-1” : { $exists : false }!}!
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – Transactions
34
{! _id : “3:/home/john/profile”,! avatar : {! “r14979e4b424-0-1” : <bin>! },! _deleted : {! “r14979e4b424-0-1” : “false”! },! _commitRoot : {! “r14979e4b424-0-1” : “1” ! }!}!
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – Transactions
35
{! _id : “3:/home/john/profile”,! avatar : {! “r14979e4b424-0-1” : <bin>,! “r14979e6c7a2-0-1” : <bin>! },! _deleted : {! “r14979e4b424-0-1” : “false”! },! _commitRoot : {! “r14979e4b424-0-1” : “1”,! “r14979e6c7a2-0-1” : “1” ! }!}!
T-1
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – Transactions
36
{! _id : “3:/home/john/profile”,! avatar : {! “r14979e4b424-0-1” : <bin>,! “r14979e6c7a2-0-1” : <bin>,! “r14979e6c7a3-0-1” : <bin>! },! _deleted : {! “r14979e4b424-0-1” : “false”! },! _commitRoot : {! “r14979e4b424-0-1” : “1”,! “r14979e6c7a2-0-1” : “1”,! “r14979e6c7a3-0-1” : “1”! }!}!
T-1 T-2
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – Transactions
37
{! _id : “1:/home”,! _deleted : { ! “r14979e1b312-0-1” : “false”! },! _revisions : {! “r14979e1b312-0-1” : “c”! },! _collisions : {! “r14979e6c7a2-0-1” : “true”! }!}!
T-2
Conditional update for collision marker: {! _id : “1:/home”, ! “_revisions.r14979e6c7a2-0-1” : { $exists : false }!}!
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – Transactions
38
{! _id : “1:/home”,! _deleted : { ! “r14979e1b312-0-1” : “false”! },! _revisions : {! “r14979e1b312-0-1” : “c”! },! _collisions : {! “r14979e6c7a2-0-1” : “true”! }!}!
T-2
Conditional update for commit: {! _id : “1:/home”, ! “_collisions.r14979e6c7a2-0-1” : { $exists : false }!}!
T-1
✗
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – Transactions
39
{! _id : “1:/home”,! _deleted : { ! “r14979e1b312-0-1” : “false”! },! _revisions : {! “r14979e1b312-0-1” : “c”,! “r14979e6c7a3-0-1” : “c”! },! _collisions : {! “r14979e6c7a2-0-1” : “true”! }!}!
T-2 T-1
✗
{! _id : “1:/home”, ! “_collisions.r14979e6c7a3-0-1” : { $exists : false }!}!
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – Transactions
40
{! _id : “3:/home/john/profile”,! avatar : {! “r14979e4b424-0-1” : <bin>,! “r14979e6c7a2-0-1” : <bin>,! “r14979e6c7a3-0-1” : <bin>! },! _deleted : {! “r14979e4b424-0-1” : “false”! },! _commitRoot : {! “r14979e4b424-0-1” : “1”,! “r14979e6c7a2-0-1” : “1”,! “r14979e6c7a3-0-1” : “1”! }!}!
T-1
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Contents
Adobe Experience Manager
Java Content Repository
Apache Jackrabbit Oak
Multiversion Concurrency Control
Transactions
Content Addressable Storage
Q & A
41
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – Content Addressable Storage
42
GridFS
file chunk 0
chunk 1
chunk N
…
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – Content Addressable Storage
43
Oak
Binary Storage Node Storage
Binary
Chunk 0x38a7
Chunk 0xc92a
Chunk 0x8f91
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – Content Addressable Storage
44
Oak
Binary Storage
Binary 0x38a7
0x38a7
Chunk 0x38a7
Chunk 0xc92a
Chunk 0x8f91
Hash:
Node Storage
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Binary
Apache Jackrabbit Oak – Content Addressable Storage
45
Oak
Binary Storage
0x8f91
Chunk 0x38a7
Chunk 0xc92a
Chunk 0x8f91
Hash:
0x38a7 0x8f91
Node Storage
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Binary
Apache Jackrabbit Oak – Content Addressable Storage
46
Oak
Binary Storage
0x52f1
Chunk 0x38a7
Chunk 0xc92a
Chunk 0x8f91
Hash:
0x38a7 0x8f91
Chunk 0x52f1
0x52f1
Node Storage
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – Content Addressable Storage
47
Oak
Binary Storage
Chunk 0x38a7
Chunk 0xc92a
Chunk 0x8f91
Chunk 0x52f1
Node Storage
0x38a7 0x8f91 0x52f1
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Apache Jackrabbit Oak – Content Addressable Storage
48
⊕
⊖
De-duplication on chunk level Chunks are immutable Shared Storage
Garbage collection
© 2014 Adobe Systems Incorporated. All Rights Reserved.
Summary
Adobe Experience Manager
Java Content Repository
Apache Jackrabbit Oak
Multiversion Concurrency Control
Transactions
Content Addressable Storage
Q & A
49
© 2014 Adobe Systems Incorporated. All Rights Reserved. 50
Q & A
© 2014 Adobe Systems Incorporated. All Rights Reserved.