fault tolerance (i). topics r basic concepts r physical redundancy r information redundancy r timing...

Fault Tolerance (I)

Topics

Basic concepts Physical Redundancy Information Redundancy Timing Redundancy RAID

Readings

Tannenbaum: 7.1,7.2

Introduction

A characteristic feature of distributed systems that distinguishes them from single-machine systems is the notion of partial failure

A partial failure may happen when one component in a distributed system fails.

This failure may affect the proper operation of other components, while at the same time leaving other components unaffected.

Introduction

An important goal in design is to construct the system in such a way that it can automatically recover from partial failures without seriously affecting the overall performance.

The distributed system should continue to operate in an acceptable way while repairs are being made.

By the way…. Computing systems are not very

reliable OS crashes frequently (Windows), buggy

software, unreliable hardware, software/hardware incompatibilities

Until recently: computer users were “tech savvy”

• Could depend on users to reboot, troubleshoot problems

By the way…. Computing systems are not very

reliable (cont) Growing popularity of Internet/World Wide

Web• “Novice” users • Need to build more reliable/dependable systems

Example: what if your TV (or car) broke down every day?

• Users don’t want to “restart” TV or fix it (by opening it up)

Need to make computing systems more reliable

Characterizing Dependable Systems

Dependable systems are characterized by Availability

• This refers to the percentage of time system may be used immediately

Reliability• Mean time to failure (MTTF) I.e., mean time

between failures. Safety

• How serious is the impact of a failure Maintainability

• How long does it take to repair the system Security

Characterizing Dependable Systems

Availability and reliability are not the same thing.

If a system goes down for a millisecond every hour, it has an availability of over 99.9999 percent, but it is still highly unreliable.

A system that never crashes but is shut down for two weeks every August has high reliability but only 96 percent availability.

Definitions

A system fails when it does not perform according to its specification.

An error is part of a system state that may lead to a failure.

A fault is the cause of an error.

Definitions

Types of Faults Transient

• Occur once and then disappear.• If the operation is repeated, the fault goes away.• Example: Bird flying through the beam of a

microwave transmitter may cause lost bits on some network (not to mention a roasted bird).

Definitions

Types of Faults (continued) Intermittent

• Occurs and then vanishes of its own accord, then reappears, etc;

• A loose connector will often cause a intermittent fault.

Permanent• Continues to exist until the faulty component is

repaired.• Burnt-out chips, software bugs, and disk head

crashes.

A fault tolerant system does not fail in the presence of faults.

Server Failure Models

Type of failure Description

Crash failure A server halts, but is working correctly until it halts

Omission failure Receive omission Send omission

A server fails to respond to incoming requestsA server fails to receive incoming messagesA server fails to send messages

Timing failure A server's response lies outside the specified time interval

Response failure Value failure State transition failure

The server's response is incorrectThe value of the response is wrongThe server deviates from the correct flow of control

Arbitrary failure A server may produce arbitrary responses at arbitrary times


Crash Failure (fail-stop) A server halts, but is working correctly until

it halts. Example: An OS that comes to a grinding

halt and for which there is only one solution: reboot


Omission Failure This occurs when a server fails to respond to

incoming requests or fails to receive incoming messages or fails to send messages.

There are many reasons for an omission failure including:

• The connection between a client and a server has been correctly established, but there was no thread to listen to incoming requests.

• A send buffer overflows; The server may need to be prepared that the client will reissue its previous request.

• An infinite loop where each iteration causes a forked process.


Timing Failures A server’s response lies outside the

specified time interval. An e-commerce site may state that the

response to a user should be no more than 5 seconds (actually this is too long).

In a video-on-demand application, the client is to receive frames at 25 frames per second give or take 2 frames.

Timing failures are very difficult to deal with.


Response Failure A server’s response is incorrect: a wrong

reply to a request is returned or when a server reacts unexpectedly to an incoming request.

Example: A search engine that systematically returns web pages not related to any of the used search terms.

Example: A server receives a message that it cannot recognize.


Arbitrary (Byzantine) Failures Arbitrary failures occur Server is producing output it should never

have produced, but which cannot be detected as being incorrect.

A faulty server may even be maliciously working together with other servers to produce intentionally wrong answers.


Ideally, we want fail-stop processes. A fail-stop process will simply stop

producing output in such a way that its halting can be detected by other processes.

The server may be so friendly to announce it is about to crash.

The reality is that processes are not that friendly.

We rely on other processes to detect the failure.


Problem: How to tell the difference between a process that has halted and a process that is just slow. Timeouts are great but theoretically you

cannot place an exact time on when to expect a response.

If most of the time, the timeout interval is too high then you are delaying the system from reacting to the failure.

Failure Masking by Redundancy

If a system is to be fault tolerant, the best it can do is to try to hide the occurrence of failures from other processes.

Key technique: Use redundancy Types of redundancy

Information redundancy Physical redundancy Time redundancy

Physical Redundancy

Extra equipment or processes are added to make it possible for the system as a whole to tolerate the loss or malfunctioning of some components.

Physical redundancy can thus be done in either hardware or in software.

Examples in hardware: Aircraft: 747’s have 4 engines but fly on 3. Space shuttle: Has 5 computers Electronic circuits

Physical Redundancy

Triple modular redundancy.

Physical Redundancy

For electronic circuits, each device is replicated three times.

Following each stage in the circuit is a triplicated voter.

Each voter is a circuit that has three inputs and one output.

If two or three of the inputs are the same, the output is equal to that input.

If all three inputs are different, the output is undefined.

This kind of design is known as TMR (Triple Modular Redundancy).

Physical Redundancy

TMR can be applied to any hardware unit.

The TMR can completely mask the failure of one hardware unit.

No explicit actions need to be performed for error detection, recovery, etc;

Particularly suitable for transient failures if we assume the basic TMR scheme (one voter, three replicas).

Physical Redundancy

This scheme can’t handle the failure of two units.

Once an unit fails, it is essential that both units should continue to work correctly.

The TMR scheme depends critically on the voting element. The voting element is typically a simple circuit and highly reliable circuits of this complexity can be built.

The failure of a single voter cannot be tolerated.

Physical Redundancy

The TMR approach can be generalized to replicating N units. This is called the NMR approach.

The larger N is then the higher the number of faults that can be completely masked.

Physical Redundancy

The basic TMR/NMR scheme is often complemented with sparing.

Sparing is often referred to as stand-by redundancy since the redundant or spare units usually are not operating online.

The restoring organ for sparing is a switch.

An error detector is also required to determine when the on-line unit has failed.

Failed units may be replaced by a spare.

Physical Redundancy

Some reliability results: Overall reliability decreases when the

degree of redundancy is increased above a certain amount.

TMR provides the least potential for reliability improvement.

NMR systems with spares provide the highest reliability.

Information Redundancy

Coding is often used in information redundancy.

Coding has been extensively used for improving the reliability of communication.

The basic idea is to add check bits to the information bits such that errors in some bits can be detected, and if possible corrected.

The process of adding check bits to information bits is called encoding.

The reverse process of extracting information from the encoded data is called decoding.


Detectability/Correctability of a Code A code defines a set of words that are

possible for that code. The Hamming distance of a code is the

minimum number of bit positions in which any two words in the code differ.

If d is the Hamming distance, D is the number of bit errors that it can detect and C is the number of bit errors it can correct, then the following relation is always true:

d = C +D +1 with D C

Information RedundancyDetectability/Correctability of a Code Let’s say that you have a code that looks like this:

000001010011100101110111

Hamming distance is one. You can’t detect an error. Why? Let’s say that a fault transforms 001 to 011.

How do you know this is a fault vs the possibility that 011 is correct?


Detectability/Correctability of a Code On the otherhand, let’s say that you have

the following code of 3 codewords:0000 0011 1100If a fault changes one bit in a correct word it will

result in a word that is not in the above list. This is not true if two bits are changed. Hence, the above code can only tolerate one fault.

You can’t correct. Let’s say that 0000 changes to 0010. You know there is an error, but how do you know that this should go to 0000 and not 0011.


Simple Parity Bits Simple parity bits have been in common use

in computer systems for many years. The parity bit is selected so that the total

number of 1’s in the codeword is odd (even) for an odd-parity (even-parity) code.

This means that the Hamming distance is 2. The parity bit can only detect single bit

errors.


Simple Parity Bits Example (assume odd-parity):

Codeword is 000; the parity bit is 1 Codeword is 001; the parity bit is 0 Codeword is 010; the parity bit is 0 Let’s say that 000 is transferred as 0001.

The parity bit is set as 1 which results in a odd number of ones (remember we are only interested in an odd number of ones).


Simple Parity Bits All errors involving an odd number of

bits can be detected because such errors will produce an incorrect parity.


Hamming Codes Multiple parity bits are added such that

each parity bit is a parity of a subset of information bits. The code can detect and also correct errors.

Widely used in semiconductor memory and in disk arrays.


Hamming Codes Parity bits occupy the bit positions

1,2,4,…. (power of 2) in the encoding. The remaining are the data positions.

Let k be the number of parity bits. Let m be the number of data bits. The word length of the encoded word is

m+k.


Hamming Codes – Example Let k = 3 and m = 4 Bits in positions 1,2,4 are the parity bits. Label

these as c1,c2 and c3. Bits in positions 3,5,6,7 are the data bits.

Label these as d1,d2,d3 and d4. The value of parity bits is defined by the

following relations:c1 = d1d2d4 c2 = d1d3d4c3 = d2d3d4

0 000 4 100

1 001 d1 5 101 d2

2 010 6 110 d3

3 011 7 111 d4


Hamming Codes – Example Let the word to be transmitted be 1011. 001 010 011 100 101 110 111

c1 c2 d1 c3 d2 d3 d4

0 1 1 0 0 1 1

c1 = d1d2d4 c2 = d1d3d4c3 = d2d3d4


Hamming Codes – Example How do we come up with these relations? A Hamming code generator computes the

check bits according to the following scheme. • The binary representation of the position number

j is jk-1 ... j1 j0.

• The value of a check bit is chosen to give odd (or even) parity over all bit positions j such that ji = 1.

• Thus each bit of the data word participates in several different check bits.


Hamming Codes – Example Assume the word transferred is 1111.

c1 c2 d1 c3 d2 d3 d4

0 1 1 0 1 1 1

001 010 011 100 101 110 111

Transmitted improperly; wasoriginally a zero.


Hamming Codes – Example Location of bits in error

The check bits obtained from the relationship give above are XORed with the actual check bits obtained from the code.

c1’ = d1d2d4 = 1c2’ = d1d3d4 = 1c3’ = d2d3d4 = 1e1 = c1c1’ = 0 1 = 1 e2 = c2c2’ = 1 1 = 0e3 = c3c3’ = 0 1 = 1

Correction is done by simply complementing the bit.

If each error bit is 0, no error; else the error location bits specify the location of the bit in error

d2 is common to c1’ and c3’ as well as c1 and c3.


Hamming Codes – Example The use of Hamming codes becomes

more efficient, in terms of numbers of bits needed relative to the number of data bits, as the word size increases.

If the data word length is 8 bits, the number of check bits will be 4. This overhead is 50%.

If the word length is 84 bits, the number of check bits will be 7 giving an overhead of 9 percent.


Cyclic Redundancy Code (CRC) These codes are applied to a block of

data, rather than independent words. CRCs are commonly used in detecting

errors in data communication. A sequence of bits is represented as a

polynomial (generator polynomial).


Cyclic Redundancy Code (CRC) If the kth bit is 1, then the polynomial contains

xk. Example:1100101101

x9 + x8 + x5 + x3 + x2 + 1

Encoding To the data bit sequence, add (k+1) bits in the end. The extended data sequence is divided (modula 2)

by the generator polynomial. The final remainder is added to the data sequence to

form the encoded data.


Cyclic Redundancy Code (CRC) Decoding

The extra (k+1) bits are just discarded to obtain the original data bits.

Error checking: The data bits are again divided by the generator polynomial and the final remainder is checked with last (k+1) bits of the received data.

If there is a difference, an error has occurred.


Cyclic Redundancy Code (CRC) Through proper selection of the generating polynomial

CRC codes will: Detect all single bit errors in the data stream Detect all double bit errors in the data stream Detect any odd number of errors in the data stream Detect any burst error for which the length of the

burst is less than the length of the generating polynomial

Detect most all larger burst errors

Time Redundancy

An action is performed and if the need arises, it is performed again.

Example: If a transaction aborts, it can be redone with no harm.

This is especially useful when the faults are transient or intermittent.

Case Study

Let’s look at RAID(Redundant Array of Inexpensive Disks).

Motivation Improve disk access time by using arrays of

disks Disks are getting inexpensive. Lower cost disks:

• Less capacity.• But cheaper, smaller, and lower power.

Disk Organization 1

Interleaving disks. Supercomputing applications. Transfer of large blocks of data at high

rates.

...

Grouped read: single read spread over multiple disks

Disk Organization 1

What is interleaving? Assume you have 4 disks. Byte interleaving means that byte N is on

disk (N mod 4). Block interleaving means that block N is on

disk (N mod 4). All reads and writes involve all disks, which

is great for large transfers

Disk Organization 2

Independent disks. Transaction processing applications. Database partitioned across disks. Concurrent access to independent items.

...

Read Write

Problem: Reliability

Disk unreliability causes frequent backups.

Fault tolerance is needed, otherwise disk arrays are too unreliable to be useful.

RAID: Use of extra disks containing redundant information. Similar to redundant transmission of data.

RAID Levels

Different levels provide different reliability, cost, and performance.

The mean time to failure (MTTF) is a function of total number of disks, number of data disks in a group (G), number of check disks per group (C), and number of groups.

The number C is determined by RAID level.

First RAID Level

Mirrors Most expensive approach. All disks duplicated (G=1 and C=1). Every write to data disk results in write to

check disk. Reads can be from either disk. Double cost and half capacity.

Second RAID Level

Data is split at the bit level and spread over data and redundancy (check) disks.

Redundant bits are computed using Hamming code and placed in the redundancy disk.

Interleave data across disks in a group. Add enough check disks to detect/correct

error. Single parity disk detects single error. Makes sense for large data transfers. Small transfers mean all disks must be

accessed (to check if data is correct).

Third and Fourth RAID Level

The third RAID level is similar to the second RAID level except that splitting of data is at the byte level. There is one parity disk.

The fourth RAID level is similar to the third RAID level except that splitting of data is at the block level. There is one parity disk

The fifth RAID level is similar to the fourth RAID level except that check bits are distributed across multiple disks.

There are 8 RAID levels.

Process Resilience

The key approach to tolerating a faulty process is to organize several identical processes in a group.

Design issues include the following: When a message is sent to the group itself,

all members of the group receive it. Dealing with process groups

Problems of Agreement A set of processes need to agree on a value

(decision), after one or more processes have proposed what that value (decision) should be

Examples: mutual exclusion, election, transactions

Processes may be correct, crashed, or they may exhibit arbitrary (Byzantine) failures

Messages are exchanged on an one-to-one basis, and they are not signed

Problems of Agreement

The general goal of distributed agreement algorithms is to have all the nonfaulty processes reach consensus on some issue and to establish that consensus within a finite number of steps.

What if processes exhibit Byzantine failures.

This is often compared to armies in the Byzantine Empire in which there many conspiracies, intrigue and untruthfulness were alleged to be common in ruling circles.

The Two-Army Problem How can two perfect processes reach

agreement about 1 bit of information ? … over an unreliable communication channel

Red army: 5000 troops Blue army #1, #2: 3000 troops each

How can the blue armies reach agreement on when to attack ?

Their only means of communication is by sending messengers

… that may be captured by the enemy ! No solution!

The Two-Army Problem Proof by contradiction: Assume there is a

solution with a minimum number of messages Suppose commander of blue army 1 is General

Alexander and the command of the blue army 2 is General Bonaparte.

General Alexander sends a message to General Bonaparte reading “I have a plan; let’s attack at dawn tomorrow”.

The messenger gets through and Bonaparte sends him back a message with a note saying “Splendid idea, Alex. See you at dawn tomorrow.”

The messenger gets back.

The Two-Army Problem Proof by contradiction (cont)

Alexander wants to make sure that Bonaparte does know that the messenger got back safely so that Bonaparte is confident that Alexander will attack.

Alexander tells the messenger to go tell Bonaparte that his message arrived and the battle is set.

The messenger gets through, but now Bonaparte worries that Alexander does not know if the acknowledgement got through.

Bonaparte acknowledges the acknowledgement. Etc etc etc

History Lesson: The Byzantine Empire

Time: 330-1453 AD. Place: Balkans and Modern Turkey. Endless conspiracies, intrigue, and

untruthfullness were alleged to be common practice in the ruling circles of the day.

That is: it was typical for intentionally wrong and malicious activity to occur among the ruling group. A similar occurance can surface in a DS, and is known as ‘byzantine failure’.

Question: how do we deal with such malicious group members within a distributed system?

Byzantine Generals Problem

Now assume that the communication is perfect but the processes are not.

This problem also occurs in military settings and is called the Byzantine Generals Problem.

We still have the red army, but n blue generals. Communication is done pairwise by phone; it is

instantaneous and perfect. m of the generals are traitors (faulty) and are

actively trying to prevent the loyal generals from reaching agreement by feeding them incorrect and contradictory information.

Is agreement still possible?

Byzantine Generals Problem We will illustrate by example where there are

4 generals, where one is a traitor (analogous to a faulty process).

Step 1: Every general sends a (reliable) message to every

other general announcing his troop strength. Loyal generals tell the truth. Traitors tell every other general a different lie. Example: general 1 reports 1K troops, general 2

reports 2K troops, general 3 lies to everyone (giving x, y, z respectively) and general 4 reports 4K troops.


Step 2: The results of the announcements of step 1

are collected together in the form of vectors.


Step 3 Consists of every general passing his vector

from the previous step to every other general.

Each general gets three vectors from each other general.

General 3 hasn’t stopped lying. He invents 12 new values: a through l.


Step 4 Each general examines the ith element of

each of the newly received vectors. If any value has a majority, that value is put

into the result vector. If no value has a majority, the

corresponding element of the result vector is marked UNKNOWN.


The same as in previous example, except now with 2 loyal generals and one traitor.


With m faulty processes, agreement is possible only if 2m+1 processes function correctly

The total is 3m+1. If messages cannot be guaranteed to be

delivered within a known, finite time, no agreement is possible if even one process is faulty.

Why? Slow processes are indistinguishable from crashed ones.


Let f be the number of faults to be tolerated.

The algorithm needs f+1 rounds. In each round, a process sends to all the

other processes the values that it received in the previous round. The number of message sent is on the order of:O(Nf+1) where N is the number of generals.

If you do not assume Byzantine faults then you need a lot less infrastructure.

fault tolerance (i). topics r basic concepts r physical redundancy r information redundancy r timing...

Documents

introduction r

r computing systems

notion of partial failure

readings r tannenbaum

definitions r types

reliable slide

security slide

omission failure