there is more consensus in egalitarian parliaments€¦ · iulian moraru, david andersen, michael...

There Is More Consensus in Egalitarian Parliaments

Iulian Moraru, David Andersen, Michael Kaminsky

Carnegie Mellon University Intel Labs

Friday, November 8, 13

Faulttolerance Redundancy⇒


State Machine Replication

3



3

Execute the same commands in the same order



3

Paxos




3

Paxos

• No external failure detector required




3

Paxos


• Fast fail-over (high availability)



Paxos is important in clustersChubby, Boxwood, SMARTER, ZooKeeper

• Synchronization• Resource discovery• Data replication

4

High throughputHigh availability


Paxos is important in the wide-areaSpanner, Megastore

• Bring data closer to clients

• Tolerate datacenter outages

5

Low latency


Paxos is important in the wide-areaSpanner, Megastore

• Bring data closer to clients

• Tolerate datacenter outages

5

Low latency

85ms130ms


Paxos overview

6


Paxos overview• Agreement protocol

6



• Tolerates F failures with 2F+1 replicas (optimal)


6





• Replicas can fail by crashing (non-Byzantine)

6





• Replicas can fail by crashing (non-Byzantine)

• Asynchronous communication

6


1 2 3 4 ...

Using Paxos to order commands

7


1 2 3 4 ...


7

A BC


1 2 3 4 ...


7

A BCvote C

vote B

vote A


1 2 3 4 ...


7

A BCvote C

vote B

vote A

ack B ack B


1 2 3 4 ...


7

A

B

C


1 2 3 4 ...


7

AB

C


1 2 3 4 ...

DUsing Paxos to order commands

7

AB

C


1 2 3 4 ...D


7

AB C


1 2 3 4 ...D


7

AB C

• Choose commands independently for each slot


1 2 3 4 ...D


7

AB C


• At least 2 RTTs per slot:


1 2 3 4 ...D


7

AB C


• At least 2 RTTs per slot:1. Take ownership of a slot


1 2 3 4 ...D


7

AB C


• At least 2 RTTs per slot:1. Take ownership of a slot

2. Propose command


1 2 3 4 ...

Multi-Paxos

8


1 2 3 4 ...

DMulti-Paxos

8

ABC


1 2 3 4 ...

DMulti-Paxos

8

A

BC


1 2 3 4 ...

DMulti-Paxos

8

A B

C


1 2 3 4 ...

DMulti-Paxos

8

A B C


1 2 3 4 ...D

Multi-Paxos

8

A B C


1 2 3 4 ...D

Multi-Paxos

8

A B C

• 1 RTT to commit


1 2 3 4 ...D

Multi-Paxos

8

A B C

• 1 RTT to commit

• Bottleneck for performance and availabilty


Can we have it all?

9


Can we have it all?

9

• High throughput, low latency


Can we have it all?

9


• Constant availability


Can we have it all?

9



• Distribute load evenly across all replicas


Can we have it all?

9




• Use fastest replicas


Can we have it all?

9





• Use closest (lowest latency) replicas


Can we have it all?

9

Paxos• High throughput, low latency






Can we have it all?

9

PaxosMulti-• High throughput, low latency






Can we have it all?

9

PaxosMulti-

Egalitarian Paxos(EPaxos)







EPaxos is all about ordering

10


EPaxos is all about orderingPrevious strategies:

10



10

...

Contend for slots



10

...

Contend for slots Paxos



10

...

Contend for slots

...

One replicadecides

Paxos



10

...

Contend for slots

...

One replicadecides

Paxos

Multi-Paxos, Fast Paxos,Generalized Paxos



10

...

Contend for slots

...

One replicadecides

Paxos


...

Take turnsround-robin



10

...

Contend for slots

...

One replicadecides

Paxos


Mencius

...

Take turnsround-robin


EPaxos intuition

11


EPaxos intuition

1 2 3 4 ...

11


EPaxos intuition

1 2 3 4 ...

A

11


EPaxos intuition

1 2 3 4 ...A

C B

11


EPaxos intuition

1 2 3 4 ...ACB

11

D


EPaxos intuition

1 2 3 4 ...ACB

11

D

After commit @ each replica


EPaxos intuition

1 2 3 4 ...ACB

11

D

A B C DAfter commit @ each replica


EPaxos intuition

1 2 3 4 ...ACB

11

• Load balance (every replica is a leader)

D



EPaxos intuition

1 2 3 4 ...ACB

11

• Load balance (every replica is a leader)

• EPaxos can choose any quorum for each command

D



EPaxos commit protocol

12

R1

R2

R3

R4

R5



12

R1

R2

R3

R4

R5

PreAccept(A)

A ➝ ∅



12

R1

R2

R3

R4

R5

PreAccept(A)

A ➝ ∅ A ➝ ∅



12

R1

R2

R3

R4

R5

PreAccept(A)

A ➝ ∅ A ➝ ∅

Commit A ➝ ∅



12

R1

R2

R3

R4

R5

PreAccept(A)

A ➝ ∅ A ➝ ∅

PreAccept(B)

B ➝ ∅

Commit A ➝ ∅



12

R1

R2

R3

R4

R5

PreAccept(A)

A ➝ ∅ A ➝ ∅

PreAccept(B)

B ➝ ∅ B ➝ ∅

Commit A ➝ ∅



12

R1

R2

R3

R4

R5

PreAccept(A)

A ➝ ∅ A ➝ ∅

PreAccept(B)

B ➝ ∅

B ➝ {A}

B ➝ ∅

Commit A ➝ ∅



12

R1

R2

R3

R4

R5

PreAccept(A)

A ➝ ∅ A ➝ ∅

PreAccept(B)

B ➝ ∅

B ➝ {A}

Accept(B)

B ➝ {A}B ➝ ∅

Commit A ➝ ∅



12

R1

R2

R3

R4

R5

PreAccept(A)

A ➝ ∅ A ➝ ∅

PreAccept(B)

B ➝ ∅

B ➝ {A}

Accept(B)

B ➝ {A}B ➝ ∅ ACK

Commit A ➝ ∅



12

R1

R2

R3

R4

R5

PreAccept(A)

A ➝ ∅ A ➝ ∅

PreAccept(B)

B ➝ ∅

B ➝ {A}

Accept(B)

B ➝ {A}B ➝ ∅ ACK

Commit A ➝ ∅

Commit B ➝ {A}



12

R1

R2

R3

R4

R5

PreAccept(A)

A ➝ ∅ A ➝ ∅

PreAccept(B)

B ➝ ∅

B ➝ {A}

Accept(B)

B ➝ {A}B ➝ ∅ ACK

Commit A ➝ ∅

Commit B ➝ {A}

PreAccept(C)

C ➝ {A}



12

R1

R2

R3

R4

R5

PreAccept(A)

A ➝ ∅ A ➝ ∅

PreAccept(B)

B ➝ ∅

B ➝ {A}

Accept(B)

B ➝ {A}B ➝ ∅ ACK

Commit A ➝ ∅

Commit B ➝ {A}

PreAccept(C)

C ➝ {A} C ➝ {A, B}



12

R1

R2

R3

R4

R5

PreAccept(A)

A ➝ ∅ A ➝ ∅

PreAccept(B)

B ➝ ∅

B ➝ {A}

Accept(B)

B ➝ {A}B ➝ ∅ ACK

Commit A ➝ ∅

Commit B ➝ {A}

PreAccept(C)

C ➝ {A} C ➝ {A, B}

Commit C ➝ {A, B}


Order only interfering commands• 1 RTT

• Non-concurrent commands

• OR non-interfering commands

• 2 RTTs

• Concurrent AND interfering

13


Interference is application-specific

14


Interference is application-specificKV store

• Infer from operation key

14




Google App Engine

• Programmer-specified

14




Google App Engine

• Programmer-specified

Relational databases

• Most transactions are simple, can be analyzed

• Few remaining transactions interfere w/ everything

14


Execution

15

A

B

D

C

E


Execution

15

A

B

D

C

E

strongly-connected component


Execution

15

A

B

D

C

E

E

D

B, C, A

1

2

3



Execution

15

A

B

D

C

E

E

D

B, C, A

1

2

3


Approximate sequence # order(Lamport clock)


EPaxos properties

16


EPaxos propertiesLinearizability: If A~B, and A committed before B proposed

then A will be executed before B.

16


EPaxos propertiesLinearizability: If A~B, and A committed before B proposed

then A will be executed before B.

16

Fast-path quorum: F +⎡F / 2⎤• Optimal for 3 and 5 replicas

• Better than Fast / Generalized Paxos by 1


Results

17


Optimal wide-area commit latency

18


85ms90ms130ms

300ms150ms


18


EPaxos

Mencius

Generalized Paxos

Multi-Paxos (CA leader)

0 50 100 150 200 250

Median Commit Latency [ms]

85ms90ms130ms

300ms150ms


18


EPaxos

Mencius

Generalized Paxos


0 50 100 150 200 250


CA

85ms90ms130ms

300ms150ms


18


VA

EPaxos

Mencius

Generalized Paxos


0 50 100 150 200 250


CA

85ms90ms130ms

300ms150ms


18


VA OR

EPaxos

Mencius

Generalized Paxos


0 50 100 150 200 250


CA

85ms90ms130ms

300ms150ms


18


VA OR

EPaxos

Mencius

Generalized Paxos


0 50 100 150 200 250


JPCA

85ms90ms130ms

300ms150ms


18


VA OR

EPaxos

Mencius

Generalized Paxos


0 50 100 150 200 250


EUJPCA

85ms90ms130ms

300ms150ms


18


VA OR

EPaxos

Mencius

Generalized Paxos


0 50 100 150 200 250


EUJPCA

85ms90ms130ms

300ms150ms


18

EPaxos: Optimal commit latency in wide-areafor 3 and 5 replicas


Operations / secOperations / sec

Multi-PaxosMencius

EPaxos 0%EPaxos 2%

EPaxos 100%

0 10000 20000 30000 40000 50000 60000

Higher + more stable throughput

19

0 10000 20000 30000 40000 50000 60000

5 replicas 3 replicas



Multi-PaxosMencius

EPaxos 0%EPaxos 100%

0 10000 20000 30000 40000 50000 60000

Multi-PaxosMencius

EPaxos 0%EPaxos 2%

EPaxos 100%

0 10000 20000 30000 40000 50000 60000

Higher + more stable throughput

19

0 10000 20000 30000 40000 50000 60000

5 replicas 3 replicas

0 10000 20000 30000 40000 50000 60000

When one replica is slow



EPaxos: higher throughput w/ batching

20

Latency[ms]

log scale

5 ms batching, local area

Multi-PaxosEPaxos 0%EPaxos 100%

Throughput (ops / sec)


EPaxos: higher throughput w/ batching

20

Latency[ms]

log scale

5 ms batching, local area

0 100000 200000 300000 400000 5000001

10

100

1000

10000Multi-PaxosEPaxos 0%EPaxos 100%

Throughput (ops / sec)


Time [sec]

0 10000 20000 30000

0 10000 20000 30000

0

10000

20000

30000

0 5 10 15 20 25 30 35

Constant availability

21

Committhroughput[ops / sec]

leader failure

replica failure

Multi-Paxos

Mencius

EPaxosreplica failure

delayed commits


EPaxos insights

22


EPaxos insights

22

Order commands explicitly


EPaxos insights

22


High throughput


EPaxos insights

22


High throughput Stability


EPaxos insights

22


High throughput Stability Low latency


EPaxos insights

22



Optimize only delays that matter(clients co-located w/ closest replica)


EPaxos insights

22



Optimize only delays that matter(clients co-located w/ closest replica)

Smaller quorums


23

TLA+ Spec

http://cs.cmu.edu/~imoraru/epaxos/tr.pdf

Formal Proof

Open Source Release

http://github.com/efficient/epaxos


http://www.cs.cmu.edu/~imoraru/epaxos/tr.pdf

http://www.cs.cmu.edu/~imoraru/epaxos/tr.pdf

https://github.com/efficient/epaxos

https://github.com/efficient/epaxos

there is more consensus in egalitarian parliaments€¦ · iulian moraru, david andersen, michael...

Documents