formalise’16 toward rigorous design of · snapshot/resume (to rewind, try other schedules) full...
TRANSCRIPT
![Page 1: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/1.jpg)
FormaliSE’16
Toward Rigorous Design of Domain-Specific Distributed Systems
Mohammed S. Al-MahfoudhGanesh GopalakrishnanRyan StatesmanThe University of Utah
![Page 2: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/2.jpg)
Outline❖ Intro
❖ Nowadays
❖ situation
❖ solutions: difficulties + effectiveness
❖ DS2
❖ offers
❖ example
❖ completion status
❖ Conclusion
![Page 3: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/3.jpg)
Intro❖ Distributed Systems gone mainstream
❖ Data centers, cloud, IoT,…etc.
❖ Notoriously hard to develop+get right
❖ Reasoning? barely supported
❖ more productivity + less reasoning =>
❖ Worse? no semantic clarityImage credit: www.scorpionpictureguide.com => cute bug is parallel processing, scorpion DS
![Page 4: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/4.jpg)
Background
❖ Extreme non-determinism
❖ Common Misconceptions
❖ fast access, single time frame, fault-freedom, strong-ordering
❖ Sadly, distributed systems violate all these!
❖ Language generality/imprecision
❖ Domain specific knowledge often not exploited
![Page 5: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/5.jpg)
–From Pamela Zave’s Talk
This morning’s lecture, you saw it!how much effort, time, and dedication it takes
![Page 6: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/6.jpg)
What does it take to specify Distributed Systems
❖ Proving Raft Linearizability in Verdi
❖ 45K of lines in complete proof
❖ 90 non-trivial invariants
❖ 3 man-years to achieve! (2 ppl x 1.5 yrs = 3)
❖ I had a kid + another coming + many things < 3 yrs!
❖ How many LoC actual Raft implementation?Complete story in [3]
![Page 7: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/7.jpg)
Well Known Issues, Current Approaches
❖ Only good for stable systems
❖ During development needs
❖ exploration (loose ends)
❖ Visualization (improving understanding)
❖ Basic Property Checking (e.g. Linearizability)
❖ Not scalable (previous slide)
❖ Not widely known in mainstream community
![Page 8: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/8.jpg)
Current success storiesDSLs: DeLite, P, P#, …etc (Domain Specific Languages)
❖ Domain implicits exploitation (case specifics handled)
❖ Clear syntax and semantics (concise+familiar)
❖ Highly optimized runnable(s) (Delite)
❖ Multiple backends (heterogeneity handled - Delite)
❖ High level language (Scala - Delite, C#-P#)
❖ No (networked) distributed systems support!
![Page 9: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/9.jpg)
DS2 Infrastructure
Domain Specific Distributed Systems Specification and Synthesis
![Page 10: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/10.jpg)
DS2 Infrastructure (Provides/Enables)
❖ Actor driven model (easy to understand)
❖ Semantically guided exploration/testing of distributed systems
❖ Extensibility, Compose-ability and re-use of algorithms
❖ Multiple levels (layers) of (non-)faulty operation
❖ Visualization of schedules/traces (understanding aids)
❖ Ultimately, Synthesis of dependable distributed systems
![Page 11: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/11.jpg)
More advantages
❖ One front-end
❖ All that framework taken care of (for all developers)
❖ No fluctuation: a model/proof vs. implementation
❖ Implementation is its own model
❖ no more separate model/proof activities.
![Page 12: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/12.jpg)
Extra Features❖ Snapshot/Resume (to rewind, try other schedules)
❖ Full runtime capture
❖ Traces untouched (keeping exploration history)
❖ Tracing Builtin (FULL state capture)
❖ For Scheduler: debugging aid
❖ For Distributed System: Analysis and Visualization
❖ Visualizer/stepper being built!
![Page 13: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/13.jpg)
Limitations
![Page 14: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/14.jpg)
Limitations
❖ Programming-Language specific
❖ Current implementation => specific to Scala
❖ Targeting Akka first (checking + synthesis)
❖ Infrastructure ported
❖ Schedulers ported
❖ front-end(s) re-written
![Page 15: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/15.jpg)
Teaser (What if — one rule takes care of code)
![Page 16: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/16.jpg)
One rule - rules them all
replicated[main][s1,s2][primary](d).on(3 updates)
![Page 17: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/17.jpg)
One rule - rules them all
replicated[main][s1,s2][primary](d).on(3 updates)
d = 0 // data item
cd = 0 // count of updates to ’d’
vd = 0 // version ID of ’d’
csd = d.hashCode() // check−sum of ’d’
replicatedOn = {d: [s1,s2],...}
alive−agents = [s1,s2]
![Page 18: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/18.jpg)
One rule - rules them all
replicated[main][s1,s2][primary](d).on(3 updates)
d = 0 // data item
cd = 0 // count of updates to ’d’
vd = 0 // version ID of ’d’
csd = d.hashCode() // check−sum of ’d’
replicatedOn = {d: [s1,s2],...}
alive−agents = [s1,s2]
cd++; vd++; csd += d.hashCode()
if (cd%3 == 0) { m = Message("Replicate", payload = [d, vd]); ds.send(main, m, s1);
ds.send(main, m, s2)
}
![Page 19: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/19.jpg)
One rule - rules them all
replicated[main][s1,s2][primary](d).on(3 updates)
d = 0 // data item
cd = 0 // count of updates to ’d’
vd = 0 // version ID of ’d’
csd = d.hashCode() // check−sum of ’d’
replicatedOn = {d: [s1,s2],...}
alive−agents = [s1,s2]
cd++; vd++; csd += d.hashCode()
if (cd%3 == 0) { m = Message("Replicate", payload = [d, vd]); ds.send(main, m, s1);
ds.send(main, m, s2)
}
// ’d ’ was updated ; recvr needs to catchup
if (m.payload(3) > recvr.vd) // just one batch update happened if(recMsg.payload(2) − recvr.vd ==3) update(recvr.locals , recMsg) // > 1 batch update , recvr missed >= 1 update
else if (recMsg.payload(2) − recvr.vd >3 )
updateElaborated ( recvr , recMsg ) // recvr ahead, let other’s know else if (recMsg.payload(2) − recvr.vd < 0 )
{ m = Message("Replicate", payload = [d,vd, csd]);
replicateTo(replicatedOn, m)}else // more sophisticated fault−tolerance work
somethingIsWrong (m) / / use checksum+others (raft)
![Page 20: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/20.jpg)
Architecture+Lang. Design
![Page 21: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/21.jpg)
Communication Patterns & Events
❖Send (communication)
❖Fire and forget message send
❖Ask (communication+synchronization)
❖Fire and return handle to (optionally) block on later/immediately
❖Handle is a (Future) object.
❖LOCK/UNLOCK (event)
❖model network partition
❖ Primitives differ from parallel programming (list on next slide)
![Page 22: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/22.jpg)
DS2 - Kinds of Events
A set of all agents
M message type
B basic block of code (to execute)
C 2 M⇥A ! K ⇥ Bstatement type (plus hidden meta data)
K 2 {none, send, ask, resolve, create, start,stop, kill, lock, unlock, stop� consume,
resume� consume, become, unbecome, stash,
unstash, unstash� all, get, get� timed,
bootstrap, bootstrap� all,modify � state}
![Page 23: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/23.jpg)
![Page 24: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/24.jpg)
we need ONE model
representing ALL
![Page 25: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/25.jpg)
Process (shared mem.) we need ONE model
representing ALL
![Page 26: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/26.jpg)
Process (shared mem.)
Threads (shared mem.)
we need ONE model
representing ALL
![Page 27: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/27.jpg)
Process (shared mem.)
Threads (shared mem.)
we need ONE model
representing ALL
What more?!PL’s Mem. Models
![Page 28: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/28.jpg)
Process (shared mem.)
Threads (shared mem.)
Actors(No Shared
mem. + comm.)
we need ONE model
representing ALL
What more?!PL’s Mem. Models
![Page 29: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/29.jpg)
Process (shared mem.)
Threads (shared mem.)
MPI Process (shared mem. + Comm.)
Actors(No Shared
mem. + comm.)
we need ONE model
representing ALL
What more?!PL’s Mem. Models
![Page 30: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/30.jpg)
Process (shared mem.)
Threads (shared mem.)
MPI Process (shared mem. + Comm.)
Event-Driven Threads(shared mem. + Events)
Actors(No Shared
mem. + comm.)
we need ONE model
representing ALL
What more?!PL’s Mem. Models
![Page 31: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/31.jpg)
Process (shared mem.)
Threads (shared mem.)
MPI Process (shared mem. + Comm.)
Event-Driven Threads(shared mem. + Events)
Actors(Some with Shared
mem. + comm.)
Actors(No Shared
mem. + comm.)
we need ONE model
representing ALL
What more?!PL’s Mem. Models
![Page 32: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/32.jpg)
Process (shared mem.)
Threads (shared mem.)
MPI Process (shared mem. + Comm.)
Event-Driven Threads(shared mem. + Events)
Replicated State Machines
(shared mem. + Events + Transitions)
Actors(Some with Shared
mem. + comm.)
Actors(No Shared
mem. + comm.)
we need ONE model
representing ALL
What more?!PL’s Mem. Models
![Page 33: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/33.jpg)
Process (shared mem.)
Threads (shared mem.)
MPI Process (shared mem. + Comm.)
Event-Driven Threads(shared mem. + Events)
Replicated State Machines
(shared mem. + Events + Transitions)
Actors(Some with Shared
mem. + comm.)
Actors(No Shared
mem. + comm.)
we need ONE model
representing ALL
What more?!PL’s Mem. Models
![Page 34: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/34.jpg)
DS2 Architecture- an Agent
Process (shared mem.)
Threads (shared mem.)
MPI Process (shared mem. + Comm.)
Event-Driven Threads(shared mem. + Events)
Replicated State Machines
(shared mem. + Events + Transitions)
Actors(Some with Shared
mem. + comm.)
Actors(No Shared
mem. + comm.)
we need ONE model
representing ALL
What more?!PL’s Mem. Models
![Page 35: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/35.jpg)
DS2 Architecture- an Agent
A single process model with: Self contained state, communication,Behaviors,other helper functions.Accommodating all kinds of processes.
Process (shared mem.)
Threads (shared mem.)
MPI Process (shared mem. + Comm.)
Event-Driven Threads(shared mem. + Events)
Replicated State Machines
(shared mem. + Events + Transitions)
Actors(Some with Shared
mem. + comm.)
Actors(No Shared
mem. + comm.)
we need ONE model
representing ALL
What more?!PL’s Mem. Models
![Page 36: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/36.jpg)
DS2 Architecture - A Strategy on a Context
Scheduler+DistributedSystemStrategy OO Design PatternScheduler = StrategyDist. Sys = ContextSimple, extensible, effective separation of concerns
![Page 37: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/37.jpg)
DS2 Architecture - Semantic-aware scheduling
Inter-related entities in a Strategy OO Design PatternScheduler = StrategyDist. Sys = ContextSimple, extensible, effective separation of concerns
![Page 38: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/38.jpg)
Example driven benefit illustration (Animated from FMI paper)
![Page 39: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/39.jpg)
High level exampleEcho Server-client interaction:
1. Server => started (bootstrapped) => unlocked
2. Client => started => unlocked => send request => waits confirmation
3. Server => process request => sends confirmation
4. Client => is happy
Scenarios:
❖ No bugs schedule (above)
❖ Deadlock 1
❖ Deadlock 2
![Page 40: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/40.jpg)
Example
val ds = new DistributedSystem("Echo-ack")
val s = new Agent("Server")
val c = new Agent("Client")
val act1, act2, act3 = new Action
// Client setup
act1 + Statement(UNLOCK,c) // unlocks the agent incoming q
act1 + Statement(ASK,c,new Message("Show","Hello!"),s, "vn")
act1 + Statement(GET,c,"vn","vn2")
act1 + Statement(println("I'm Happy!"))
c.R("Start") = act1 // (Start, act1) to reactions map
// Server setup
act2 + Statement(UNLOCK, s)
act2 + Statement(println("Greetings!"))
act3 + Statement((m:Message,a:Agent)=>println(m.p))
act3 + Statement((m:Message,a:Agent)=>send(s,m(p = true),m.s))
s.R("Start") = act2 ; s.R("Show") = act3
ds += Set(s,c) // adding agents to system
ds.attach(BasicScheduler)
![Page 41: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/41.jpg)
Correct Schedulesch.consume(s) // consume resolving send(..) stmt
// note GET blocks, then it is resolved
sch.consume(c) // consume "happy" stmt from c-task
sch.executeOne // s print("Hello")
sch.executeOne // c blocks on GET, doesn't progress
// putting back all stmts after it
// from cq back to front of task.xq in order
sch.executeOne // resolving send(..), t.q != empty
// things happen to t.L("vn")-future resolved
// and then c.q = [RF(f,s=s)], note sender
// is s, not t
sch.handel(c) // handling the RF message, unblocking c
sch.consume(c) // consuming GET from c again
sch.consume(c) // consuming "happy" stmt from c
sch.executeOne // R-GET c-stmt, won't block (resolved)
// c.L("vn2") = c.L("vn").val
sch.executeOne // print("I'm happy")
// DONE happy schedule, other schedules are not this happy
val sch = ds.scheduler
sch.boot(s); sch.boot(c) // sends Start msg to s and to c
sch.schedule(s) // schedule start-task from s
sch.schedule(c) // schedule start-task from c
sch.consume(s) // consume UNLOCK stmt from s-task
sch.consume(s) // consume "greeting" stmt from s-task
sch.consume(c) // consume UNLOCK stmt from c-task
sch.consume(c) // consume ASK stmt from c-task
sch.executeOne // UNLOCK s-stmt, IsLocked(s) == false
sch.executeOne // "greeting" s-stmt
sch.executeOne // UNLOCK c-stmt, IsLocked(c) == false
sch.executeOne // ASK s-stmt, T = {t} temporary agent
// and s.q == [Show("Hello",s=t)]
sch.schedule(s) // schedule "Show" task from s
sch.consume(s) // consume print("Hello") stmt
sch.consume(c) // consume GET stmt from c-task
![Page 42: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/42.jpg)
animated schedule
Initial state (nothing executed)
To Execute:
sch.boot(s)
sch.boot(c)
![Page 43: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/43.jpg)
animated schedule
Executed:
sch.boot(s)
sch.boot(c)
To Execute:
sch.schedule(s)
sch.schedule(c)
![Page 44: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/44.jpg)
animated scheduleExecuted:
sch.schedule(s)
sch.schedule(c)
To Execute:
sch.consume(s)
sch.consume(s)
sch.consume(c)
sch.consume(c)
![Page 45: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/45.jpg)
animated scheduleExecuted:
sch.consume(s)
sch.consume(s)
sch.consume(c)
sch.consume(c)
To Execute:
sch.executeOne
sch.executeOne
sch.executeOne
![Page 46: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/46.jpg)
animated schedule
Executed:
sch.executeOne
sch.executeOne
sch.executeOne
To Execute:
sch.executeOne // ask stmt
![Page 47: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/47.jpg)
animated schedule
Executed:
sch.executeOne
sch.executeOne
sch.executeOne
To Execute:
sch.executeOne // ask stmt
![Page 48: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/48.jpg)
animated scheduleExecuted:
sch.executeOne // ask stmt
To Execute:
sch.schedule(s) // "Show" task
sch.consume(s) // print("Hello")
sch.consume(c) // consume GET
sch.consume(s) // consume r-send
// note GET blocks, then it is resolved
sch.consume(c) // consume "happy"
![Page 49: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/49.jpg)
After some time …
![Page 50: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/50.jpg)
animated schedule
Executed:
sch.executeOne // r-send(..)
To Execute:
sch.handel(c) // RF
![Page 51: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/51.jpg)
animated schedule
Executed:
sch.handel(c) // RF
To Execute:
sch.consume(c) // GET
sch.consume(c) // "happy" stmt
![Page 52: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/52.jpg)
animated schedule
Executed:
sch.consume(c) // GET
sch.consume(c) // "happy" stmt
To Execute:
sch.executeOne // R-GET
![Page 53: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/53.jpg)
animated schedule
Executed:
sch.executeOne // R-GET
To Execute:
sch.executeOne //"I'm happy"
![Page 54: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/54.jpg)
animated schedule
Executed:
sch.executeOne //"I'm happy"
To Execute:
![Page 55: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/55.jpg)
What could have gone wrong?
![Page 56: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/56.jpg)
May Go Wrong
❖ Client could have blocked first
❖ Before server resolves: it crashes => deadlock
❖ After server resolves: RF dropped => deadlock
❖ Messages in Agent’s queue are still in-flight
❖ Till they are handled/stashed, then delivered
❖ Both avoidable by timed-get on future.
![Page 57: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/57.jpg)
Deadlock1 Schedule (dropped resolve future msg)
![Page 58: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/58.jpg)
About to drop a message!
Executed:
sch.executeOne // r-send(..)
To Execute:
sch.handel(c) // RF
![Page 59: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/59.jpg)
RF message dropped!
Executed:
simulated-RF-msg-drop
To Execute:
![Page 60: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/60.jpg)
Deadlock2 Schedule (crashed server before resolve)
![Page 61: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/61.jpg)
Client is blocked
Executed:
sch.executeOne // c blocks
To Execute:
sch.executeOne // r-send(..)
![Page 62: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/62.jpg)
Server about to resolve but…
Executed:
sch.executeOne // c blocks
To Execute:
sch.executeOne // r-send(..)
![Page 63: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/63.jpg)
Server crashed before resolve …
Executed:
simalted-crash
server-came-back (empty hand)
To Execute:
![Page 64: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/64.jpg)
That simple example taught us: “more erroneous interleaving
than correct ones!”
![Page 65: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/65.jpg)
Completion Status
![Page 66: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/66.jpg)
Implementation/Completion Status
completednot started partial completion / in progressstarted
DS2 Lang. Spec.
DS2 model (shown here)
Chord, Zab, Multi-Paxos, Raft
Tracing
Basic Scheduler
Akka front-end
Snapshot/Resume Linearizability Sch.
DS2 Lang. impl.
Synthesis
Visualization
![Page 67: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/67.jpg)
Conclusion
![Page 68: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/68.jpg)
Conclusion❖ Motivated the need for an integrated solution
❖ Presented our model
❖ How it solves the issues stated
❖ Walk through example(s)
❖ Sneak peak towards synthesis
❖ Future work: Formal Operational Semantics (under review), Tool for Akka (with multiple alg.), Synthesis of Akka from DS2.
![Page 69: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/69.jpg)
References
[1] ”Toward Rigorous Design of Domain-Specific Distributed Systems”, Mohammed S. Al-Mahfoudh, Ganesh Gopalakrishnan, Ryan Stutsman.
[2] http://formalverification.cs.utah.edu/ds2/
[3] ”Planning for Change in a Formal Verification of the Raft Consensus Protocol”, Doug Woos, Zachary Tatlock, James R. Wilcox, Michael D. Ernst, Steve Anton, Thomas Anderson.
![Page 70: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/70.jpg)
Q/A
![Page 71: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/71.jpg)
Thank you!
![Page 72: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/72.jpg)
Removed frames follow
![Page 73: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/73.jpg)
animated schedule
Executed:
sch.schedule(s) // "Show" task
sch.consume(s) // print("Hello")
sch.consume(c) // consume GET
sch.consume(s) // consume r-send
sch.consume(c) // "happy"
To Execute:
sch.executeOne // s print("Hello")
![Page 74: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/74.jpg)
animated schedule
Executed:
sch.executeOne // print("Hello")
To Execute:
sch.executeOne // c blocks
![Page 75: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/75.jpg)
animated schedule
Executed:
sch.executeOne // c blocks
To Execute:
sch.executeOne // r-send(..)
![Page 76: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/76.jpg)
animated schedule
Executed:
sch.executeOne // c blocks
To Execute:
sch.executeOne // r-send(..)
![Page 77: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/77.jpg)
animated schedule
Executed:
sch.executeOne // c blocks
To Execute:
sch.executeOne // r-send(..)
![Page 78: FormaliSE’16 Toward Rigorous Design of · Snapshot/Resume (to rewind, try other schedules) Full runtime capture Traces untouched (keeping exploration history) Tracing Builtin (FULL](https://reader033.vdocuments.net/reader033/viewer/2022050423/5f920083290070032b559ec2/html5/thumbnails/78.jpg)
animated schedule
Executed:
sch.executeOne // c blocks
To Execute:
sch.executeOne // r-send(..)