failures & redundancy - zoo | yale...

Failures & Redundancy

Ennan ZhaiComputer Science at Yale University

ennan.zhai@yale.edu

About our labs, final project and midterm

• Building a simple and elegant Peerster• Making git log clear• Midterm may be kind of hard

• Lab1 grading

About our labs, final project and midterm

• Building a simple and elegant Peerster• Making git log clear• Midterm may be kind of hard

• Lab1 grading

About our labs, final project and midterm

• Building a simple and elegant Peerster• Making git log clear• Midterm may be kind of hard

• Lab1 grading

About our labs, final project and midterm

• Building a simple and elegant Peerster• Making git log clear• Midterm

• Lab1 grading

About our labs, final project and midterm

• Building a simple and elegant Peerster• Making git log clear• Midterm

• Lab1 grading

• Failures

• Correlated failures in decentralized systems

Lecture Outline

• Failures

• Correlated failures in decentralized systems

Lecture Outline

• What is the failure?• Why?• Real evidences?

• Partial failure

Failures

• What is the failure?• Why?• Real evidences?

• Partial failure

Failures

• What is the failure?• Why?• Real evidences?

• Partial failure

Failures

Evidence

Category) Failure)types) Diagnosis)&)Repair)

%)

So#ware(21%( Link(layer(loop( Find(and(fix(bugs(

19%(Imbalance(!(overload( 2%(

Hardware(18%( FCS(error( Replace(cable( 13%(Unstable(power( Repair(power( 5%(

Unknown(23%( Switch(stops(forwarding( N/A( 9%(Imbalance(!(overload( 7%(Lost(configuraNon( 5%(High(CPU(uNlizaNon( 2%(

ConfiguraNon(38%(

Errors(on(mulNple(switches(

Update(configuraNon(

32%(

Errors(on(one(switch( 6%(

Evidence

Category) Failure)types) Diagnosis)&)Repair)

%)

So#ware(21%( Link(layer(loop( Find(and(fix(bugs(

19%(Imbalance(!(overload( 2%(

Hardware(18%( FCS(error( Replace(cable( 13%(Unstable(power( Repair(power( 5%(

Unknown(23%( Switch(stops(forwarding( N/A( 9%(Imbalance(!(overload( 7%(Lost(configuraNon( 5%(High(CPU(uNlizaNon( 2%(

ConfiguraNon(38%(

Errors(on(mulNple(switches(

Update(configuraNon(

32%(

Errors(on(one(switch( 6%(

Evidence

Category) Failure)types) Diagnosis)&)Repair)

%)

So#ware(21%( Link(layer(loop( Find(and(fix(bugs(

19%(Imbalance(!(overload( 2%(

Hardware(18%( FCS(error( Replace(cable( 13%(Unstable(power( Repair(power( 5%(

Unknown(23%( Switch(stops(forwarding( N/A( 9%(Imbalance(!(overload( 7%(Lost(configuraNon( 5%(High(CPU(uNlizaNon( 2%(

ConfiguraNon(38%(

Errors(on(mulNple(switches(

Update(configuraNon(

32%(

Errors(on(one(switch( 6%(

Evidence

Category) Failure)types) Diagnosis)&)Repair)

%)

So#ware(21%( Link(layer(loop( Find(and(fix(bugs(

19%(Imbalance(!(overload( 2%(

Hardware(18%( FCS(error( Replace(cable( 13%(Unstable(power( Repair(power( 5%(

Unknown(23%( Switch(stops(forwarding( N/A( 9%(Imbalance(!(overload( 7%(Lost(configuraNon( 5%(High(CPU(uNlizaNon( 2%(

ConfiguraNon(38%(

Errors(on(mulNple(switches(

Update(configuraNon(

32%(

Errors(on(one(switch( 6%(

Evidence

Top10 Failure Events in Clouds

Failure Models

• Crash failure• Timing failure• Response failure

• Byzantine failure

Failure Models

• Crash failure• Timing failure• Response failure

• Byzantine failure

Failure Models

• Crash failure• Timing failure• Response failure

• Byzantine failure

Failure Models

• Crash failure• Timing failure• Response failure

• Byzantine failure

Failure Models

Solutions

Solutions

• Overcoming failures after the outage occurs:- Diagnosis system- Accountability system- Fault tolerance system

Solutions

• Overcoming failures after the outage occurs:- Diagnosis system- Accountability system- Fault tolerance system

Solutions

• Overcoming failures after the outage occurs:- Diagnosis system- Accountability system- Fault tolerance system

Solutions

• Overcoming failures after the outage occurs:- Diagnosis system- Accountability system- Fault tolerance system

• Overcoming failures before the outage occurs:- Redundancy

• Failures

• Correlated failures in decentralized systems

Lecture Outline

• Failures

• Correlated failures in decentralized systems

Lecture Outline

Example

Email App

Peer A Peer B

Example

Email App

Peer A Peer B

Third-party infrastructure components

Example

Email App

ISP BISP A ISP C

Third-party infrastructure components

Peer A Peer B

Example

Email App

ISP BISP A ISP C

Power Source

Third-party infrastructure components

Peer A Peer B

Example

Email App

ISP BISP A ISP C

Power Source

Third-party infrastructure components

Peer A Peer B

Example

Email App

ISP BISP A ISP C

Power Source

Third-party infrastructure components

Peer A Peer B

Become unavailable !

Example

Example

• Service providers allocate or tolerate failures via: - diagnosis systems, e.g., Sherlock.- fault-tolerant systems, e.g., F10, Skute.

Existing Efforts

• Service providers allocate or tolerate failures via: - diagnosis systems, e.g., Sherlock.- fault-tolerant systems, e.g., F10, Skute.

• Solving the problem after the outage occurs

• There is no any effort before the problem occur

Existing Efforts

• Service providers allocate or tolerate failures via: - diagnosis systems, e.g., Sherlock.- fault-tolerant systems, e.g., F10, Skute.

• Solving the problem after the outage occurs

• We want to prevent the problem before the outage occurs

Existing Efforts

• Service providers allocate or tolerate failures via: - diagnosis systems, e.g., Sherlock.- fault-tolerant systems, e.g., F10, Skute.

• Solving the problem after the outage occurs

• We want to prevent the problem before the outage occurs

• Recommending truly independent redundancy services when deploying applications

Existing Efforts

What kind of system we want to build?

Consumer

Node A Node B Node C

Select two nodes for redundancy

Node A Node B Node C

Consumer

A and B ?

Node A Node B Node C

Consumer

B and C ?

Node A Node B Node C

Consumer

A and C ?

Node A Node B Node C

Consumer

Recommender

Select two nodes for redundancy: A&B? B&C? or A&C?

Node A Node B Node C

Consumer

Recommender

Node A Node B Node C

Consumer

Recommender

Assessing independence by the # of overlapping components between nodes

Node A Node B Node C

Consumer

Recommender

Node A Node B Node C

Consumer

Recommender

ISP A Power BPower A

Node A Node B Node C

Consumer

Recommender

ISP A Power BPower A

ISP APower APower B

Node A Node B Node C

Consumer

Recommender

ISP A Power BPower A

ISP APower APower B

Node A Node B Node C

Consumer

Recommender

ISP APower APower B

ISP A Power B ISP BPower A

Node A Node B Node C

Consumer

Recommender

ISP APower APower B

ISP A Power B ISP BPower A

ISP BPower APower B

Node A Node B Node C

Consumer

Recommender

ISP APower APower B

ISP A Power B ISP BPower A

ISP BPower APower B

Node A Node B Node C

Consumer

Recommender

ISP APower APower B

ISP BPower APower B

ISP A Power B ISP B Power CPower A

Node A Node B Node C

Consumer

Recommender

ISP BPower APower B

ISP A Power B ISP B Power CPower A

ISP BPower C

Node A Node B Node C

ISP APower APower B

Consumer

Recommender

Cloud A, C 0 Cloud B, C 1 Cloud A, B 2

Deployment | |

ISP BPower APower B

ISP A Power B ISP B Power CPower A

ISP BPower C

Node A Node B Node C

ISP APower APower B

Consumer

Recommender

ISP A Power B ISP B Power CPower A

Cloud A, C 0 Cloud B, C 1 Cloud A, B 2

Deployment | |

ISP BPower APower B

ISP BPower C

Node A Node B Node C

ISP APower APower B

Consumer

Recommender

ISP APower APower B

ISP BPower APower B

ISP A Power B ISP B Power CPower A

ISP BPower C

Cloud A, C 0 Cloud B, C 1 Cloud A, B 2

Deployment | |

=2

Node A Node B Node C

Consumer

Recommender

ISP APower APower B

ISP BPower APower B

ISP A Power B ISP B Power CPower A

ISP BPower C

Cloud A, C 0 Cloud B, C 1 Cloud A, B 2

Deployment | |

=2

Deployment | |

Node A Node B Node C

Node A, C 0 Node B, C 1 Node A, B 2

Consumer

Recommender

ISP APower APower B

ISP BPower APower B

ISP A Power B ISP B Power CPower A

ISP BPower C

Cloud A, C 0 Cloud B, C 1 Cloud A, B 2

Deployment | |Deployment | |

Node A Node B Node C

Node A, C 0 Node B, C 1 Node A, B 2

Consumer

Recommender

ISP APower APower B

ISP BPower APower B

ISP A Power B ISP B Power CPower A

ISP BPower C

Cloud A, C 0 Cloud B, C 1 Cloud A, B 2

Deployment | |Deployment | |

=1Node A Node B Node C

Node A, C 0 Node B, C 1 Node A, B 2

Consumer

Recommender

ISP APower APower B

ISP BPower APower B

ISP A Power B ISP B Power CPower A

ISP BPower C

=1

Deployment | |

Node A Node B Node C

Node A, C 0 Node B, C 1 Node A, B 2

Consumer

Recommender

ISP APower APower B

ISP A Power B ISP B Power CPower A

ISP BPower C

Deployment | |

Node A Node B Node C

Node A, C 0 Node B, C 1 Node A, B 2

Consumer

Recommender

ISP APower APower B

ISP A Power B ISP B Power CPower A

ISP BPower C

Deployment | |

=0

Node A Node B Node C

Node A, C 0 Node B, C 1 Node A, B 2

Consumer

Recommender

ISP APower APower B

ISP A Power B ISP B Power CPower A

ISP BPower C

Deployment | |

=0

Node A Node B Node C

Node A, C 0 Node B, C 1 Node A, B 2

Consumer

Recommender

Node A, C 0 Node B, C 1 Node A, B 2

Deployment | |

Node A Node B Node C

Consumer

Recommender1. Node A, C 02. Node B, C 13. Node A, B 2

| |Deployment

Ranking List

Node A Node B Node C

Consumer

But, it is not so easy .

Recommender

Node A

Node B

Node C

Solution 1

Consumer

Recommender

App Provider

Node A

Node B

Node C

Privacy Concern!

Solution 1

Trusted Third Party

Node A

Node B

Node C

Solution 2

Consumer

Trusted Third Party

App Provider

Node A

Node B

Node C

Hard to find!

Solution 2

Secure Multiparty Computation

Node A

Node B

Node C

Solution 3

Consumer

Secure Multiparty Computation

App Provider

Node A

Node B

Node C

SMPC is difficult to scale!

Solution 3

Intersection cardinality does help

Example

Our solution - iRec

• The first independence recommender sys: - achieving our goal- preserving privacy of each node- practical

Our solution - iRec

• The first independence recommender sys: - achieving our goal- preserving privacy of each node- practical

Preliminary background: P-SOP

Our solution - iRec

• Allows k parties to compute the intersection, union cardinality and Jaccard similarity, without learning other information.

S1 S2 ... ... Sn

S1 S2 ... ... Sn

J(S1, S2, ..., Sn) =

P-SOP: Private Jaccard Similarity

| |

||

• Allows k parties to compute the intersection, union cardinality and Jaccard similarity, without learning other information.

S1 S2 ... ... Sn

S1 S2 ... ... Sn

J(S1, S2, ..., Sn) =

P-SOP: Private Jaccard Similarity

| |

||

11

3

10

1

5

20

3

7

3

P-SOP

• Allows k parties to compute the intersection, union cardinality and Jaccard similarity, without learning other information.

P-SOP: Private Jaccard Similarity

11

3

10

1

5

20

3

7

3

• Allows k parties to compute the intersection, union cardinality and Jaccard similarity, without learning other information.

One overlapping element

One overlapping element

One overlapping element

P-SOP

P-SOP: Private Jaccard Similarity

11

3

10

1

5

20

3

7

3

Protocol

• Allows k parties to compute the intersection, union cardinality and Jaccard similarity, without learning other information.

But I do not know which element is overlapping

But I do not know which element is overlapping

But I do not know which element is overlapping

P-SOP

P-SOP: Private Jaccard Similarity

11

3

10

1

5

20

3

7

3

Protocol

• Allows k parties to compute the intersection, union cardinality and Jaccard similarity, without learning other information.

7 elements in union 7 elements in union

7 elements in union

P-SOP

P-SOP: Private Jaccard Similarity

11

3

10

1

5

20

3

7

3

Protocol

• Allows k parties to compute the intersection, union cardinality and Jaccard similarity, without learning other information.

But I do not know which elements are in union

But I do not know which elements are in union

But I do not know which elements are in union

P-SOP

P-SOP: Private Jaccard Similarity

11

3

10

1

5

20

3

7

3

Protocol

• Allows k parties to compute the intersection, union cardinality and Jaccard similarity, without learning other information.

P-SOP

P-SOP: Private Jaccard Similarity

S1 S2 ... ... Sn

S1 S2 ... ... Sn

J(S1, S2, ..., Sn) = | |

||

11

3

10

1

5

20

3

7

3

Protocol

• Allows k parties to compute the intersection, union cardinality and Jaccard similarity, without learning other information.

Jaccard Jaccard

Jaccard

P-SOP

P-SOP: Private Jaccard Similarity

S1 S2 ... ... Sn

S1 S2 ... ... Sn

J(S1, S2, ..., Sn) = | |

||

37

15

203

113

10

Each party maintains a commutative encryption key

David Eve

Frank

37

15

203Kd

Kf

Ke

113

10

Each party maintains a commutative encryption key

David Eve

Frank

37

15

203Kd

Kf

Ke

113

10

Each party maintains a commutative encryption keyCommutative encryption holds: Kx(Ky(E)) = Ky(Kx(E))

David Eve

Frank

Kf(3)Kf(7)

Ke(1)Ke(5)

Ke(20)Ke(3)Kd

Kf

Ke

Kd(11)Kd(3)

Kd(10)

Each party encrypts each item of elements in its dataset through the key

David Eve

Frank

Kf(7)Kf(3)

Ke(5)Ke(3)Ke(1)

Ke(20)Kd

Kf

Ke

Kd(10)Kd(11)Kd(3)

Each party shuffles the encrypted elements in its own dataset

David Eve

Frank

Ke(5)Ke(3)Ke(1)

Ke(20)Kd

Kf

Ke

Kd(10)Kd(11)Kd(3)

Kf(7)Kf(3)

Each party sends its own encrypted dataset to its successor party.

David Eve

Frank

Ke(5)Ke(3)Ke(1)

Ke(20)

Kd

Kf

Ke

Kd(10)Kd(11)Kd(3)

Kf(7)Kf(3)

Each party sends its own encrypted dataset to its successor party.

David Eve

Frank

Kf(Ke(5))Kf(Ke(3))Kf(Ke(1))

Kf(Ke(20))

Kd

Kf

Ke

Ke(Kd(10))Ke(Kd(11))Ke(Kd(3))

Kd(Kf(7))Kd(Kf(3))

Each party encrypts each item of elements in the received dataset using its own key

David Eve

Frank

Kd(Kf(3))Kd(Kf(7))

Kf(Ke(1))Kf(Ke(20))Kf(Ke(5))Kf(Ke(3))

Kd

Kf

Ke

Ke(Kd(11))Ke(Kd(10))Ke(Kd(3))

Shuffle too.

David Eve

Frank

Kd(Kf(3))Kd(Kf(7))

Kf(Ke(1))Kf(Ke(20))Kf(Ke(5))Kf(Ke(3))

Kd

Kf

Ke

Ke(Kd(11))Ke(Kd(10))Ke(Kd(3))

Each party sends its current encrypted dataset to its successor party.

David Eve

Frank

Kd(Kf(3))Kd(Kf(7))

Kf(Ke(1))Kf(Ke(20))Kf(Ke(5))Kf(Ke(3)) Kd

Kf

Ke

Ke(Kd(11))Ke(Kd(10))Ke(Kd(3))

Each party sends its current encrypted dataset to its successor party.

David Eve

Frank

Ke(Kd(Kf(3)))Ke(Kd(Kf(7)))

Kd(Kf(Ke(1)))Kd(Kf(Ke(20)))Kd(Kf(Ke(5)))Kd(Kf(Ke(3))) Kd

Kf

Ke

Kf(Ke(Kd(11)))Kf(Ke(Kd(10)))Kf(Ke(Kd(3)))

Each party encrypts each item of elements in the received dataset using its own key

David Eve

Frank

Ke(Kd(Kf(3)))Ke(Kd(Kf(7)))

Kd(Kf(Ke(3)))Kd(Kf(Ke(5)))Kd(Kf(Ke(1)))

Kd(Kf(Ke(20))) Kd

Kf

Ke

Kf(Ke(Kd(3)))Kf(Ke(Kd(10)))Kf(Ke(Kd(11)))

Shuffle.

David Eve

Frank

Ke(Kd(Kf(3)))Ke(Kd(Kf(7)))

Kd(Kf(Ke(3)))Kd(Kf(Ke(5)))Kd(Kf(Ke(1)))

Kd(Kf(Ke(20))) Kd

Kf

Ke

Kf(Ke(Kd(3)))Kf(Ke(Kd(10)))Kf(Ke(Kd(11)))

Each party sends its current encrypted dataset to its successor party.

David Eve

Frank

Ke(Kd(Kf(3)))Ke(Kd(Kf(7)))

Kd(Kf(Ke(3)))Kd(Kf(Ke(5)))Kd(Kf(Ke(1)))

Kd(Kf(Ke(20)))Kd

Kf

Ke

Kf(Ke(Kd(3)))Kf(Ke(Kd(10)))Kf(Ke(Kd(11)))

OK. Now, each party has received its own original dataset which has been encrypted by all the parties.

David Eve

Frank

Ke(Kd(Kf(3)))Ke(Kd(Kf(7)))

Kd(Kf(Ke(3)))Kd(Kf(Ke(5)))Kd(Kf(Ke(1)))

Kd(Kf(Ke(20)))Kd

Kf

Ke

Kf(Ke(Kd(3)))Kf(Ke(Kd(10)))Kf(Ke(Kd(11)))

Kx(Ky(E)) = Ky(Kx(E))

David Eve

Frank

Ke(Kd(Kf(3)))Ke(Kd(Kf(7)))

Kd(Kf(Ke(3)))Kd(Kf(Ke(5)))Kd(Kf(Ke(1)))

Kd(Kf(Ke(20)))Kd

Kf

Ke

Kf(Ke(Kd(3)))Kf(Ke(Kd(10)))Kf(Ke(Kd(11)))

Kx(Ky(E)) = Ky(Kx(E))

David Eve

Frank

Ke(Kd(Kf(3)))Ke(Kd(Kf(7)))

Kd(Kf(Ke(3)))Kd(Kf(Ke(5)))Kd(Kf(Ke(1)))

Kd(Kf(Ke(20)))Kd

Kf

Ke

Kf(Ke(Kd(3)))Kf(Ke(Kd(10)))Kf(Ke(Kd(11)))

I know the # of intersection is 1, and union is 7

I know the # of intersection is 1, and union is 7

I know the # of intersection is 1, and union is 7

David Eve

Frank

Ke(Kd(Kf(3)))Ke(Kd(Kf(7)))

Kd(Kf(Ke(3)))Kd(Kf(Ke(5)))Kd(Kf(Ke(1)))

Kd(Kf(Ke(20)))Kd

Kf

Ke

Kf(Ke(Kd(3)))Kf(Ke(Kd(10)))Kf(Ke(Kd(11)))

Jaccard = 1/7

Jaccard = 1/7

Jaccard = 1/7

David Eve

Frank

Consumer iRec

ISP A Power B ISP B Power CPower A

Node A Node B Node C

iRec

iRec

ISP A Power B ISP B Power CPower A

Select two nodes for redundancy: A&B? B&C? or A&C?

Node A Node B Node C

iRec

Consumer

iRec

ISP A Power B ISP B Power CPower A

Node A Node B Node C

iRec: Step 1

Consumer

iRec

ISP A Power B ISP B Power CPower A

Node A Node B Node C

iRec: Step 2

Consumer

ISP A Power B ISP B Power CPower A

iRec

ISP APower APower B

ISP BPower APower B

ISP BPower C

Node A Node B Node C

iRec: Step 3

Consumer

ISP A Power B ISP B Power CPower A

iRec

ISP APower APower B

ISP BPower APower B

ISP BPower C

Node A Node B Node C

iRec: Step 3

P-SOP

Consumer

ISP A Power B ISP B Power CPower A

iRec

ISP APower APower B

ISP BPower APower B

ISP BPower C

Node A Node B Node C

iRec: Step 3

=2

Consumer

ISP A Power B ISP B Power CPower A

iRec

ISP APower APower B

ISP BPower APower B

ISP BPower C

Node A Node B Node C

iRec: Step 3

=4=2

Consumer

ISP A Power B ISP B Power CPower A

iRec

ISP APower APower B

ISP BPower APower B

ISP BPower C

Node A Node B Node C

iRec: Step 3

=4=2

J = 2/4

Consumer

ISP A Power B ISP B Power CPower A

iRec

ISP APower APower B

ISP BPower APower B

ISP BPower C

Node A Node B Node C

iRec: Step 3

=4=2

J = 2/4

Cloud A, C 0 Cloud B, C 1 Cloud A, B 2

Deployment | | Cloud A, C 0 Cloud B, C 0.25 Node A, B 0.5

Deployment J

Consumer

ISP A Power B ISP B Power CPower A

iRec

ISP APower APower B

ISP BPower APower B

ISP BPower C

Node A Node B Node C

P-SOP

iRec: Step 3 Cloud A, C 0 Cloud B, C 1 Cloud A, B 2

Deployment | | Cloud A, C 0 Cloud B, C 0.25 Node A, B 0.5

Deployment J

Consumer

ISP A Power B ISP B Power CPower A

iRec

ISP APower APower B

ISP BPower APower B

ISP BPower C

Node A Node B Node C

=1

iRec: Step 3 Cloud A, C 0 Cloud B, C 1 Cloud A, B 2

Deployment | | Cloud A, C 0 Cloud B, C 0.25 Node A, B 0.5

Deployment J

Consumer

ISP A Power B ISP B Power CPower A

iRec

ISP APower APower B

ISP BPower APower B

ISP BPower C

Node A Node B Node C

=1=4

iRec: Step 3 Cloud A, C 0 Cloud B, C 1 Cloud A, B 2

Deployment | | Cloud A, C 0 Cloud B, C 0.25 Node A, B 0.5

Deployment J

Consumer

ISP A Power B ISP B Power CPower A

iRec

ISP APower APower B

ISP BPower APower B

ISP BPower C

Node A Node B Node C

=1=4

J = 1/4

iRec: Step 3 Cloud A, C 0 Cloud B, C 1 Cloud A, B 2

Deployment | | Cloud A, C 0 Cloud B, C 0.25 Node A, B 0.5

Deployment J

Consumer

ISP A Power B ISP B Power CPower A

iRec

ISP APower APower B

ISP BPower APower B

ISP BPower C

Node A Node B Node C

=1=4

J = 1/4

iRec: Step 3 Cloud A, C 0 Cloud B, C 1 Cloud A, B 2

Deployment | | Cloud A, C 0 Node B, C 0.25 Node A, B 0.5

Deployment J

Consumer

ISP A Power B ISP B Power CPower A

iRec

ISP APower APower B

ISP BPower C

Node A Node B Node C

iRec: Step 3 Cloud A, C 0 Cloud B, C 1 Cloud A, B 2

Deployment | | Cloud A, C 0 Node B, C 0.25 Node A, B 0.5

Deployment J

P-SOP

Consumer

ISP A Power B ISP B Power CPower A

iRec

ISP APower APower B

ISP BPower C

Node A Node B Node C

iRec: Step 3 Cloud A, C 0 Cloud B, C 1 Cloud A, B 2

Deployment | | Cloud A, C 0 Node B, C 0.25 Node A, B 0.5

Deployment J

=0,

Consumer

ISP A Power B ISP B Power CPower A

iRec

ISP APower APower B

ISP BPower C

Node A Node B Node C

iRec: Step 3 Cloud A, C 0 Cloud B, C 1 Cloud A, B 2

Deployment | | Cloud A, C 0 Node B, C 0.25 Node A, B 0.5

Deployment J

=0 =5 ,,

Consumer

ISP A Power B ISP B Power CPower A

iRec

ISP APower APower B

ISP BPower C

Node A Node B Node C

iRec: Step 3 Cloud A, C 0 Cloud B, C 1 Cloud A, B 2

Deployment | | Cloud A, C 0 Node B, C 0.25 Node A, B 0.5

Deployment J

=0 =5 J = 0/5,,

Consumer

ISP A Power B ISP B Power CPower A

iRec

ISP APower APower B

ISP BPower C

Node A Node B Node C

iRec: Step 3 Cloud A, C 0 Cloud B, C 1 Cloud A, B 2

Deployment | | Node A, C 0 Node B, C 0.25 Node A, B 0.5

Deployment J

=0 =5 J = 0/5,,

Consumer

iRec

ISP A Power B ISP B Power CPower A

Node A Node B Node C

iRec: Step 4 Cloud A, C 0 Cloud B, C 1 Cloud A, B 2

Deployment | | Node A, C 0 Node B, C 0.25 Node A, B 0.5

Deployment J

Consumer

ISP A Power B ISP B Power CPower A

iRec

Node A Node B Node C

iRec: Step 5 Cloud A, C 0 Cloud B, C 1 Cloud A, B 2

Deployment | | Node A, C 0 Node B, C 0.25 Node A, B 0.5

Deployment J

Consumer

Ranking List

ISP A Power B ISP B Power CPower A

iRec

Node A Node B Node C

iRec: Step 6

Cloud A, C 0 Cloud B, C 1 Cloud A, B 2

Deployment | | Node A, C 0 Node B, C 0.25 Node A, B 0.5

Deployment JConsumer

Failure is a very important topic

It is very hard to solve in decentralized system

Thanks!

Questions?