offline first: application data and synchronization
TRANSCRIPT
Offline-firstapplication data and synchronization18 February 2016
⌘
Maksym Barbul
2
Maksym Barbul
Senior Software Engineermaxbarbul [at]
gmail.com
Introduction
3
Problem statement
4
So, why is it a problem?
● user is blocked in offline● connection (blink, detection delay,
unstable)● data is always changingIt may bring user to loose data 5
What is different?
6
● Client UX (non-blocking, responsive, reliable)
● Client has smart caching● Client partially duplicates server’s logic● Controlled connectivity● Smart synchronization● Client-side security and audit
Offline-first is a Paradigmis a distinct set of concepts or thought patterns, including theories, research methods, postulates, and standards for what constitutes legitimate contributions to a field 7
§
Who needs these super-smart apps?
● Field workers● Store observation● Technical audit● Medical
observationB
US
INE
SS
EFFICIENCY8
Who needs these super-smart apps?
● Instant messengers● Social networks● Document editors● Media organizers● GTD toolsA
LL U
SE
RS
LOYALTY9
Offline Application
10
Components of an offline-first app
11
Troubleshooting
User Experience
Offline Data
Connectivity
Synchronization
Offline First
How is an offline-first app built?
12
Offline data
Synchronization
Connectivity
User Experience
Trouble-shooting
Offline Data
13
Data slices
14
Application
Static Meta-Data
BusinessData
System Runtime Data
Data slices: Application
15
Application
Static Meta-Data
BusinessData
System Runtime Data
Core (scripts)Modules (scripts)Styles & Views (scripts, CSS)Media resources
Data slices: Static Meta-Data
16
Application
Static Meta-Data
BusinessData
System Runtime Data
Configuration & SettingsLayout DeclarationPermissionsTranslations
Data slices: Business Data
17
Application
Static Meta-Data
BusinessData
System Runtime Data
DictionariesFetched Data (Read only)Entered Data (User Data)
Data slices: System Runtime Data
18
Application
Static Meta-Data
BusinessData
System Runtime Data
Logs (action, error, profiling)Session DataTransaction States (Sync)
Data Synchronization
19
Data Synchronization Types
20
Data Synchronization Types
21
Replication
Synchronization (replication) Strategies
22
● Pessimistic - no inconsistency● Optimistic - eventual consistency
Eventual Consistency
means that replicas eventually reach the same value if clients stop submitting updates
23
§
Optimistic replication algorithm
24
1.Operation submission (user submission)2.Propagation to other replicas3.Scheduling of operations to apply by
each replica4.Conflict resolution (syntactic, semantic)5.Commitment on a final schedule and
conflicts resolution result
Strong Eventual Consistency (SEC)
is a property of some eventually-consistent systems: replicas that have received and applied the same set of updates must immediately have equivalent state
25
§
Conflict-free replicated data type (CRDT)
26
is a type of specially-designed data structure used to achieve strong eventual consistency (SEC) and monotonicity (absence of rollbacks)
● operation-based
● state-based
Operation-based CRDTs (CmRDT)
27
commutative replicated data types
The operations are commutative, so can be received and applied in any order; however, they are not idempotent, and additional network protocol guarantees are required to ensure unique delivery
a * b = b * a
O(a) ≠ O(O(a))
State-based CRDTs (CvRDT)
28
convergent replicated data types
CvRDTs send their full local state to other replicas
State-based CRDTs (CvRDT): Interface
29
➔ query - reads the state of the replica, with no side effects
➔ update - writes to the replica state in accordance with certain restrictions (monotonically increase)
➔ merge - merges local state with the state of some remote replica (commutative, associative, and idempotent)
a * b = b * aO(a, b) = O(O(a,
b), b)
a*(b*c) = (a*b)*c
CvRDT: increment only
30
is an element offor each i
CvRDT: grow-only set (add)
31
add {e} to set A
S.A is a subset of T.A
unite sets
CvRDT: 2P-set (add, remove)
32
Operational transformation (OT)
33
is a technology for supporting a range of collaboration functionalities in advanced collaborative software systems. OT was originally invented for consistency maintenance and concurrency control in collaborative editing of plain text documentsApach
e Wave
Google Docs
Operational transformation (OT)
34
➔ O1 = Insert[0, "x"] // “xabc”(to insert character "x" at position "0")
➔ O2 = Delete[2, "c"] // “xabc” ???(to delete the character "c" at position "2")
“abc”
➔ * O2 = Delete[3, "c"] // “xab”
Operational transformation (OT)
35
Unfortunately, implementing OT sucks. There's a million algorithms with different tradeoffs, mostly trapped in academic papers. The algorithms are really hard and time consuming to implement correctly. ... Wave took 2 years to write and if we rewrote it today, it would take almost as long to write a second time.Joseph Gentle
who is an ex Google Wave engineer and an author of the Share.JS
Example: Chat
36
{String} title
{Object} participants
{Array} messages
CvRDT increment-only (LWW)
CvRDT set
CmRDT or CvRDT
Example: Booking
37
{Date} date_from,
date_to
{String} user_id
{String} room_id
{String} booking_status
CvRDT (LWW)
NONE (not editable)
CvRDT (LWW)
CvRDT
Example: Collaborative field editing
38
{String} textOperational
transformation
Don’t forget about AUDIT
39
●Why do we have this value?●Who did this change?●When?●What was the previous value?
References: read theory● Offline First manifesto http://offlinefirst.org/● Links to articles, events, samples https://github.com/offlinefirst● Designing Offline-First Web Apps http://alistapart.com/article/offline-first ● SAY HELLO TO OFFLINE FIRST http://hood.ie/blog/say-hello-to-offline-first.html ● Optimistic replication https://www.wikiwand.com/en/Optimistic_replication ● CRDT http://www.wikiwand.com/en/Conflict-free_replicated_data_type ● OT http://www.wikiwand.com/en/Operational_transformation ● appSync.org: open-source patterns & code for data synchronization in mobile apps
http://confluence.tapcrowd.com/pages/viewpage.action;jsessionid=3F4D2C44DBFC46644A7955F82A416DC2?pageId=2262404
● * Free Book “Distributed Systems” by Mikito Takada http://book.mixu.net/distsys/index.html● Roshi CRDT explanation
https://developers.soundcloud.com/blog/roshi-a-crdt-system-for-timestamped-events● A comprehensive study of CRDTs by Marc Shapiro
https://hal.inria.fr/file/index/docid/555588/filename/techreport.pdf
40
References: source code● appSync source https://bitbucket.org/nikonelissen/appsync/src ● CRDT Roshi by SoundCloud with great expanation https://github.com/soundcloud/roshi ● CRDT Riak by Basho (League of Legends) https://github.com/basho/ ● ShareJS https://github.com/share/ShareJS ● Hoodie https://github.com/hoodiehq/hoodie
41
References: borrowed resources● Sync types diagram http://confluence.tapcrowd.com/pages/viewpage.action?pageId=2262404 ● Free icons found here http://findicons.com/
42
Summary: how to build offline-first apps
43
● Know the Paradigm● Plan data structure and
synchronization approaches before development
● Be paranoid about user’s data● Develop offline-first applications
The End?
44
Thank you.eager to use your great Offline
apps
😉
Questions...
45
🙋