![Page 1: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/1.jpg)
TRUST Spring Conference, April 2-3, 2008
Write Markers for Probabilistic Quorum Systems
Michael Merideth, Carnegie Mellon University
Michael Reiter, University of North Carolina
![Page 2: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/2.jpg)
4/3/08 Michael Merideth 2
Replication via Quorum Systems
Replicated data– Server becomes n replicas
Server
Clients
![Page 3: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/3.jpg)
4/3/08 Michael Merideth 3
Replicas Replicated data
– Server becomes n replicas Clients issue read and write
operations– Involve quorums (subsets)
of replicas
High availability– Yet, no writes lost,
forged, or corrupted Clients
Replication via Quorum Systems
![Page 4: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/4.jpg)
4/3/08 Michael Merideth 4
Types of Servers (in Examples)
bowling ball ice cream fish any value
non-faulty faulty
![Page 5: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/5.jpg)
4/3/08 Michael Merideth 5
Types of Clients (in Examples)
non-faulty faulty
![Page 6: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/6.jpg)
4/3/08 Michael Merideth 6
Write Operation
Client wants to write “ice cream” to system
![Page 7: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/7.jpg)
4/3/08 Michael Merideth 7
Write Operation
Client submits write to write quorum
![Page 8: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/8.jpg)
4/3/08 Michael Merideth 8
Write Operation Complete
Positive responses from quorum means write complete
![Page 9: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/9.jpg)
4/3/08 Michael Merideth 9
Write Operation Complete
![Page 10: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/10.jpg)
4/3/08 Michael Merideth 10
Read Operation
Client queries read quorum for values
![Page 11: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/11.jpg)
4/3/08 Michael Merideth 11
Read Operation
Determines read value based on votes (responses) from entire quorum
(Chooses “ice cream”)
![Page 12: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/12.jpg)
4/3/08 Michael Merideth 12
Write Markers Concept
Write marker: additional data (written with value) that identifies write quorum– Verified by clients during read
Improves properties of probabilistic quorum systems– Tolerate more faults and use smaller quorums
![Page 13: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/13.jpg)
4/3/08 Michael Merideth 13
Outline
Strict, Byzantine quorum systems Probabilistic, Byzantine quorum systems Benefits of write markers Idea for implementation
![Page 14: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/14.jpg)
4/3/08 Michael Merideth 14
Byzantine Quorum System[malkhi & reiter 98]
Byzantine (arbitrary) faults– Faulty nodes may lie– Faulty clients and servers may
collude b faulty servers
– Identity of faulty nodes unknown by non-faulty nodes
![Page 15: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/15.jpg)
4/3/08 Michael Merideth 15
Write Operation
Write quorum may contain faulty servers
![Page 16: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/16.jpg)
4/3/08 Michael Merideth 16
Write Operation Complete
![Page 17: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/17.jpg)
4/3/08 Michael Merideth 17
Read Operation
Faulty servers may fabricate value
![Page 18: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/18.jpg)
4/3/08 Michael Merideth 18
Stale Values
Stale (logically older) values are detectible
![Page 19: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/19.jpg)
4/3/08 Michael Merideth 19
Conflicting Values
Faulty servers may also fabricate conflicting (logically concurrent) values– E.g., same timestamp
Here “fish” conflicts with “ice cream”– But ice cream has more votes
![Page 20: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/20.jpg)
4/3/08 Michael Merideth 20
More Conflicting Values
Non-faulty servers may also return conflicting values
For example, in single-round write protocols– Such protocols are desirable for
efficiency– Client may (perhaps unknowingly)
submit a write that is conflicting
![Page 21: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/21.jpg)
4/3/08 Michael Merideth 21
Conflicting Write
Same as normal write
![Page 22: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/22.jpg)
4/3/08 Michael Merideth 22
Conflicting Write Incomplete
Accepted by non-faulty servers that have not accepted (conflicting) value
Write does not complete
![Page 23: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/23.jpg)
4/3/08 Michael Merideth 23
Which Value is Correct?
“Ice cream” was complete– … therefore is correct
“Fish” was incomplete– … therefore should be ignored
But ice cream and fish get equal votes
Client uncertain?
![Page 24: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/24.jpg)
4/3/08 Michael Merideth 24
Conflicting Values: Problematic
Must outvote conflicting replicas Thus, many potentially conflicting
replicas implies ability to tolerate (relatively) few faults
?
![Page 25: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/25.jpg)
4/3/08 Michael Merideth 25
Impact of Conflicting Replicas
Quorum Conflict Faults Protocols
Opaque< n/5 (least)
e.g., Q/U
Masking < n/4e.g., Fleet,
PASIS
Dissemin-ation
< n/3 (most)
e.g., BFT, HQ
?
![Page 26: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/26.jpg)
4/3/08 Michael Merideth 26
Choice of Quorums Important
Choices of read quorum and both write quorums led to problem– Other choices lead to correct
answer
?
![Page 27: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/27.jpg)
4/3/08 Michael Merideth 27
Choice of Quorums Important
Choices of read quorum and both write quorums led to problem– Other choices lead to correct
answer
![Page 28: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/28.jpg)
4/3/08 Michael Merideth 28
Idea: Select Quorums at Random
In fact, correct answer in expectation (in this example)– If quorums chosen uniformly at
random(an access strategy)
![Page 29: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/29.jpg)
4/3/08 Michael Merideth 29
Probabilistic Quorum Systems[malkhi, reiter, wool, wright 01]
Weakening intersection property to hold only with high probability – Provides better availability – Tolerates more faults
Bounds error probability– Probability that quorums chosen
according to access strategy yield incorrect (or uncertain) result
![Page 30: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/30.jpg)
4/3/08 Michael Merideth 30
Probabilistic Opaque Quorum Systems[merideth & reiter 07]
Generalize access strategy– Quorums chosen from access sets– Access sets are chosen according to
access strategy
Tolerate Byzantine clients for all probabilistic quorum systems– Enforce access strategy
![Page 31: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/31.jpg)
4/3/08 Michael Merideth 31
Probabilistic Quorum Systems
Reduce number of conflicting values in expectation– Therefore, tolerate more faults (with some bounded probability of error)
Conflicting
Faults
Strict Prob.
Opaque< n/5
(fewest)
Masking < n/4
Dissemination < n/3
< n/3.15
< n/2.62
< n (most)
![Page 32: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/32.jpg)
4/3/08 Michael Merideth 32
Reduce conflicting replicas further?
Yes (for probabilistic masking and opaque quorum systems)– Write markers
![Page 33: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/33.jpg)
4/3/08 Michael Merideth 33
Write Markers
Recall,– Write operations write values– Read operations poll replicas for values
Write marker– Additional data (written with value) that identifies
the write quorum (or access set) that was used– Client accepts vote (during read) only if replica was
part of write quorum (or access set)
![Page 34: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/34.jpg)
4/3/08 Michael Merideth 34
Write Operations with Write Markers
Create write marker for quorum
![Page 35: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/35.jpg)
4/3/08 Michael Merideth 35
Write Operation Complete
![Page 36: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/36.jpg)
4/3/08 Michael Merideth 36
Conflicting Write with Write Markers
Same as normal write
![Page 37: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/37.jpg)
4/3/08 Michael Merideth 37
Conflicting Write Incomplete
Accepted by non-faulty servers that have not accepted (conflicting) value
![Page 38: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/38.jpg)
4/3/08 Michael Merideth 38
Which Value is Correct?
“Ice cream” was complete– … therefore is correct
“Fish” was incomplete– … therefore should be ignored
![Page 39: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/39.jpg)
4/3/08 Michael Merideth 39
Which Value is Correct?
Faulty client can only vote for “triangle”
Faulty client cannot vote for “star”
![Page 40: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/40.jpg)
4/3/08 Michael Merideth 40
Benefit of Write Markers
Faulty servers cannot vote for conflicting value unless they are part of write
Due to probabilistic access strategy, faulty server not always part of write
Thus, fewer conflicting servers to outvote in expectation
![Page 41: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/41.jpg)
4/3/08 Michael Merideth 41
Benefits of Write Markers
Conflicting
Faults
Strict Prob.Write-
markers
Opaque< n/5 (fewest)
Masking < n/4
Dissemination < n/3
< n/3.15
< n/2.62
< n (most)
< n/2.62
< n/2
< n(most)
Tolerate more faults
![Page 42: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/42.jpg)
4/3/08 Michael Merideth 42
Benefits of Write Markers
Tolerate more faults Use smaller quorums
– See paper
![Page 43: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/43.jpg)
4/3/08 Michael Merideth 43
Example with Benign Clients
For writes: clients choose access sets uniformly at random – Then encode and, e.g., digitally sign their choices (i.e.,
create a write marker) For reads: clients verify write marker
![Page 44: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/44.jpg)
4/3/08 Michael Merideth 44
Write Markers with Byzantine Clients
Faulty clients:– Cannot be trusted to follow access strategy– May intentionally choose quorums that maximize
conflicting values
Constrain clients [merideth&reiter 07]– Even faulty clients follow access strategy– Avoids additional communication on critical path– Choice is verified by servers as (pseudo) random
Treat choice as write marker– Modify protocol so that clients also verify choice
![Page 45: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/45.jpg)
4/3/08 Michael Merideth 45
Protocol Intuition
Servers provide pseudorandom sequence of access sets per client– Threshold signature from servers
…
![Page 46: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/46.jpg)
4/3/08 Michael Merideth 46
Servers provide pseudorandom sequence of access sets per client– Threshold signature from servers
For each operation, client locally chooses next access set in sequence; servers verify choice
Protocol Intuition
…
![Page 47: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/47.jpg)
4/3/08 Michael Merideth 47
Protocol Intuition
…
Servers provide pseudorandom sequence of access sets per client– Threshold signature from servers
For each operation, client locally chooses next access set in sequence; servers verify choice
![Page 48: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/48.jpg)
4/3/08 Michael Merideth 48
Misuse by Faulty Client
What if faulty client:– Skips ahead to “better” access
set?– Waits to perform operation until
advantageous?
In either case, access set no longer random
…
![Page 49: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/49.jpg)
4/3/08 Michael Merideth 49
Defending Against Misuse
Exponential increase in cost to use later access sets– Client puzzle (requires solution)
Correct value propagates in background [c.f. malkhi et al. 03]
Sequence becomes invalid as system progresses– Must obtain new sequence
…
![Page 50: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/50.jpg)
4/3/08 Michael Merideth 50
Write Markers Mechanism
Use client puzzle – Servers already verify solution
Have clients verify as well – Treat solution and access set
as write marker– Return during read operations
Provides mechanism for write markers
…
![Page 51: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/51.jpg)
4/3/08 Michael Merideth 51
Conclusion
Write markers provide benefits for probabilistic quorum systems– Reduce number of faulty servers that can vote for
conflicting value in expectation– Increase number of faults that can be tolerated
Opaque: up to n/2.62 (probabilistic: n/3.15; strict: n/5) Masking: up to n/2 (probabilistic: n/2.62; strict: n/4)
– Allow for smaller quorums in some cases
For more information:– Write Markers for Probabilistic Quorum Systems.
Michael G. Merideth and Michael K. Reiter. CMU Technical Report: CMU-ISR-08-110
![Page 52: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/52.jpg)
4/3/08 Michael Merideth 52
Questions?
![Page 53: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/53.jpg)
4/3/08 Michael Merideth 53
![Page 54: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/54.jpg)
4/3/08 Michael Merideth 54
![Page 55: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/55.jpg)
4/3/08 Michael Merideth 55
![Page 56: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/56.jpg)
4/3/08 Michael Merideth 56
![Page 57: Write Markers for Probabilistic Quorum Systems](https://reader036.vdocuments.net/reader036/viewer/2022062321/56812d0e550346895d91e7af/html5/thumbnails/57.jpg)
4/3/08 Michael Merideth 57