eiger: stronger semantics for low-latency geo-replicated ... · eiger: stronger semantics for!...

23
Eiger: Stronger Semantics for Low-Latency Geo-Replicated Storage Wyatt Lloyd * Michael J. Freedman * Michael Kaminsky David G. Andersen * Princeton, Intel Labs, CMU

Upload: others

Post on 27-Feb-2021

16 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Eiger: Stronger Semantics for Low-Latency Geo-Replicated ... · Eiger: Stronger Semantics for! Low-Latency! Geo-Replicated ! Storage! Wyatt Lloyd*!" Michael J. Freedman*! "" Michael

Eiger: Stronger Semantics for !Low-Latency !

Geo-Replicated !Storage!

Wyatt Lloyd*!" Michael J. Freedman*!" " Michael Kaminsky†!" " " David G. Andersen‡!

*Princeton, †Intel Labs, ‡CMU!

Page 2: Eiger: Stronger Semantics for Low-Latency Geo-Replicated ... · Eiger: Stronger Semantics for! Low-Latency! Geo-Replicated ! Storage! Wyatt Lloyd*!" Michael J. Freedman*! "" Michael

Geo-Replicated Storage !

2!

is the backend of massive websites !

FriendOf!

FriendOf!

Like!

“Halting is Undecidable”!

Page 3: Eiger: Stronger Semantics for Low-Latency Geo-Replicated ... · Eiger: Stronger Semantics for! Low-Latency! Geo-Replicated ! Storage! Wyatt Lloyd*!" Michael J. Freedman*! "" Michael

Storage Dimensions!

A-F !

G-L!

M-R!

S-Z!

Shard Data Across Many Nodes !

3!

Like!

“Halting is Undecidable”!

FriendOf! FriendO

f!

Page 4: Eiger: Stronger Semantics for Low-Latency Geo-Replicated ... · Eiger: Stronger Semantics for! Low-Latency! Geo-Replicated ! Storage! Wyatt Lloyd*!" Michael J. Freedman*! "" Michael

Storage Dimensions!

A-F !

G-L!

M-R!

S-Z!

Shard Data Across Many Nodes !

A-F!

G-L!

M-R !

S-Z!

A-F!

G-L!

M-R!

S-Z!

Data Geo-Replicated In Multiple Datacenters!

4!

Page 5: Eiger: Stronger Semantics for Low-Latency Geo-Replicated ... · Eiger: Stronger Semantics for! Low-Latency! Geo-Replicated ! Storage! Wyatt Lloyd*!" Michael J. Freedman*! "" Michael

Sharded, Geo-Replicated Storage !

A-F !

G-L!

M-R!

S-Z!

A-F!

G-L!

M-R !

S-Z!

A-F!

G-L!

M-R!

S-Z!

5!

Page 6: Eiger: Stronger Semantics for Low-Latency Geo-Replicated ... · Eiger: Stronger Semantics for! Low-Latency! Geo-Replicated ! Storage! Wyatt Lloyd*!" Michael J. Freedman*! "" Michael

Low Latency!–  Improves user experience !– Correlates with revenue!

Strong Consistency!– Obey user expectations!– Easier for programmers!

Fundamentally in Conflict![LiptonSandberg88, AttiyaWelch94] !

Strong Consistency or Low Latency !

Page 7: Eiger: Stronger Semantics for Low-Latency Geo-Replicated ... · Eiger: Stronger Semantics for! Low-Latency! Geo-Replicated ! Storage! Wyatt Lloyd*!" Michael J. Freedman*! "" Michael

Strong Consistency or Low Latency!

Megastore [SIGMOD ‘08]!!

Spanner [OSDI ‘12] !.!!

Gemini [OSDI ‘12] !.!!

Walter [SOSP ’11] !.!

Dynamo [SOSP ’07]!

"  COPS [SOSP ’11]!

"  Eiger! Obey user expectations! Easier for programmers !

Causal+ Consistency !Rich Data Model!Read-only Txns!Write-only Txns!!

Page 8: Eiger: Stronger Semantics for Low-Latency Geo-Replicated ... · Eiger: Stronger Semantics for! Low-Latency! Geo-Replicated ! Storage! Wyatt Lloyd*!" Michael J. Freedman*! "" Michael

Eiger Ensures Low Latency!

A-F !

G-L!

M-R!

S-Z!

A-F!

G-L!

M-R !

S-Z!

A-F!

G-L!

M-R!

S-Z!

8!

Keep All Ops Local!

Page 9: Eiger: Stronger Semantics for Low-Latency Geo-Replicated ... · Eiger: Stronger Semantics for! Low-Latency! Geo-Replicated ! Storage! Wyatt Lloyd*!" Michael J. Freedman*! "" Michael

Causal+ Consistency Across DCs!•  If A happens before B!–  Everyone sees A before B!

•  Obeys user! expectations!

•  Simplifies! programming!

New Job!!

Friends !Boss!Then!

Then!

Then!

Page 10: Eiger: Stronger Semantics for Low-Latency Geo-Replicated ... · Eiger: Stronger Semantics for! Low-Latency! Geo-Replicated ! Storage! Wyatt Lloyd*!" Michael J. Freedman*! "" Michael

Causal For Column Families!

•  Operations update/read many columns !•  Range query columns concurrent w/ deletes!•  Counter columns!•  See paper for details !

Lovelace!

Turing!

Church! Lovelace! Turing!

-! -! 1/1/54!

9/1/36! 1/1/54! -!

Friends!

631!

457!

Friends! Count!Profile!

Age! Town!

197! London!

100! Princeton!

Key1!

Key2!

Val!

Val!

Page 11: Eiger: Stronger Semantics for Low-Latency Geo-Replicated ... · Eiger: Stronger Semantics for! Low-Latency! Geo-Replicated ! Storage! Wyatt Lloyd*!" Michael J. Freedman*! "" Michael

Viewing Data Consistently Is Hard!Asynchronous requests + distributed data = ????? !

A!

B!

C!

1!5!6!

Update A !2!

Update B!3!

Update C!4!

??? !

Page 12: Eiger: Stronger Semantics for Low-Latency Geo-Replicated ... · Eiger: Stronger Semantics for! Low-Latency! Geo-Replicated ! Storage! Wyatt Lloyd*!" Michael J. Freedman*! "" Michael

Read-Only Transactions!•  Logical time gives a global view of data store!–  Clocks on all nodes, carried with all messages !

•  Insight: Store is consistent at all logical times !

Logical Time !

A! 0! 2!

B! 0! 3!

C! 0! 5!

A1! A2!

B1! B2!

C1! C2!

Page 13: Eiger: Stronger Semantics for Low-Latency Geo-Replicated ... · Eiger: Stronger Semantics for! Low-Latency! Geo-Replicated ! Storage! Wyatt Lloyd*!" Michael J. Freedman*! "" Michael

Read-Only Transactions!•  Extract consistent up-to-date view of data!– Across many servers !

•  Challenges!– Scalability!

•  Decentralized algorithm!– Guaranteed low latency!

•  At most 2 parallel rounds of local reads !•  No locks, no blocking!

– High performance!•  Normal case: 1 round of reads!

13!

Page 14: Eiger: Stronger Semantics for Low-Latency Geo-Replicated ... · Eiger: Stronger Semantics for! Low-Latency! Geo-Replicated ! Storage! Wyatt Lloyd*!" Michael J. Freedman*! "" Michael

Read-Only Transactions!•  Round 1: Optimistic parallel reads !•  Calculate effective time !•  Round 2: Parallel read_at_times!

Logical Time !

A! 0! 2!

B! 0! 3!

C! 0! 5!

A1! A2!

B1! B2!

C1! C2!

Distributed Storage!Client!0! 1!

A1!

4! 6!C2!

3!B2!

5!A2!

Page 15: Eiger: Stronger Semantics for Low-Latency Geo-Replicated ... · Eiger: Stronger Semantics for! Low-Latency! Geo-Replicated ! Storage! Wyatt Lloyd*!" Michael J. Freedman*! "" Michael

Logical Time !

A! 2!

B! 3!

C! 5!

A2!

B2!

C2!

Transaction Intuition!•  Read-only transactions !–  Read from a single logical time !

•  Write-only transactions !–  Appear at a single logical time !

Bonus:!Works for

Linearizability!

A3!

B3!

C3!

Page 16: Eiger: Stronger Semantics for Low-Latency Geo-Replicated ... · Eiger: Stronger Semantics for! Low-Latency! Geo-Replicated ! Storage! Wyatt Lloyd*!" Michael J. Freedman*! "" Michael

Eiger Provides!√ Low latency!√ Rich data model!√ Causal+ consistency !√ Read-only transactions!√ Write-only transactions! But what does all this cost?! Does it scale? !

16!

Page 17: Eiger: Stronger Semantics for Low-Latency Geo-Replicated ... · Eiger: Stronger Semantics for! Low-Latency! Geo-Replicated ! Storage! Wyatt Lloyd*!" Michael J. Freedman*! "" Michael

Eiger Implementation!•  Fork of open-source Cassandra!

•  +5K lines of Java to Cassandra’s 75K !

•  Code Available:!– https://github.com/wlloyd/eiger!

Page 18: Eiger: Stronger Semantics for Low-Latency Geo-Replicated ... · Eiger: Stronger Semantics for! Low-Latency! Geo-Replicated ! Storage! Wyatt Lloyd*!" Michael J. Freedman*! "" Michael

Evaluation!•  Cost of stronger consistency & semantics !– Vs. eventually-consistent Cassandra!– Overhead for real (Facebook) workload!– Overhead for state-space of workloads !

•  Scalability!

Page 19: Eiger: Stronger Semantics for Low-Latency Geo-Replicated ... · Eiger: Stronger Semantics for! Low-Latency! Geo-Replicated ! Storage! Wyatt Lloyd*!" Michael J. Freedman*! "" Michael

A-F!

G-L!

M-R!

S-Z!

A-F!

G-L!

M-R!

S-Z!

19!

Experimental Setup!Local Datacenter (Stanford) !

Remote DC (UW)!

8! 8!

8!

Page 20: Eiger: Stronger Semantics for Low-Latency Geo-Replicated ... · Eiger: Stronger Semantics for! Low-Latency! Geo-Replicated ! Storage! Wyatt Lloyd*!" Michael J. Freedman*! "" Michael

Facebook Workload Results!

20!

!!!!!!!!!!!

6.6% Overhead!

Page 21: Eiger: Stronger Semantics for Low-Latency Geo-Replicated ... · Eiger: Stronger Semantics for! Low-Latency! Geo-Replicated ! Storage! Wyatt Lloyd*!" Michael J. Freedman*! "" Michael

Eiger Scales !

21!

Facebook Workload !

384 Machines! !

Scales out !

Page 22: Eiger: Stronger Semantics for Low-Latency Geo-Replicated ... · Eiger: Stronger Semantics for! Low-Latency! Geo-Replicated ! Storage! Wyatt Lloyd*!" Michael J. Freedman*! "" Michael

Improving Low-Latency Storage !" " " " " " COPS " " ! Eiger!

Data model " " Key-Value " " ! Column-Family!

Read-only Txns "" Causal stores" ! All stores!

Write-only Txns " None " " " ! Yes!

Performance "" " Good " " " ! Great!

DC Failure"" " " Throughput " ! Resilient!" " " " " " degradation!

Page 23: Eiger: Stronger Semantics for Low-Latency Geo-Replicated ... · Eiger: Stronger Semantics for! Low-Latency! Geo-Replicated ! Storage! Wyatt Lloyd*!" Michael J. Freedman*! "" Michael

Eiger!•  Low-latency geo-replicated storage!– Causal+ for column families !– Read-only transactions!– Write-only transactions!

•  Demonstrated in working system!– Competitive with eventual!– Scales to large clusters!– https://github.com/wlloyd/eiger!