critical system

7/21/2019 Critical System

1/45

Ian Sommerville 2004 Software Engineering, 7th edition. Chapter 3 Slide 1

Critical Systems


2/45


Objectives

To explain what is meant by a critical system

where system failure can have severe

human or economic consequence.

To explain four dimensions of dependability -availability, reliability, safety and security.

To explain that, to achieve dependability, you

need to avoid mistaes, detect and removeerrors and limit dama!e caused by failure.


3/45


Topics covered

" simple safety-critical system

System dependability

"vailability and reliability

Safety Security


4/45


Critical Systems

Safety-critical systems

# $ailure results in loss of life, injury or dama!e to the

environment%

# Chemical plant protection system%

&ission-critical systems

# $ailure results in failure of some !oal-directed activity%

# Spacecraft navi!ation system%

'usiness-critical systems

# $ailure results in hi!h economic losses%

# Customer accountin! system in a ban%


5/45


System dependability

$or critical systems, it is usually the case that the

most important system property is the dependability

of the system.

The dependability of a system reflects the user(sde!ree of trust in that system. )t reflects the extent of

the user(s confidence that it will operate as users

expect and that it will not *fail( in normal use.

+sefulness and trustworthiness are not the same

thin!. " system does not have to be trusted to be

useful.


6/45


)mportance of dependability

Systems that are not dependable and are

unreliable, unsafe or insecure may be

rejected by their users.

The costs of system failure may be veryhi!h.

+ndependable systems may cause

information loss with a hi!h consequentrecovery cost.


7/45Ian Sommerville 2004 Software Engineering, 7th edition. Chapter 3 Slide 7

evelopment methods for critical systems

The costs of critical system failure are so

hi!h that development methods may be used

that are not cost-effective for other types of

system. xamples of development methods

# $ormal methods of software development

#Static analysis

# xternal quality assurance



Socio-technical critical systems

ardware failure# ardware fails because of desi!n and

manufacturin! errors or because componentshave reached the end of their natural life.

Software failure# Software fails due to errors in its specification,

desi!n or implementation.

Operational failure# uman operators mae mistaes. /ow perhaps

the lar!est sin!le cause of system failures.



" software-controlled insulin pump

+sed by diabetics to simulate the function of

the pancreas which manufactures insulin, an

essential hormone that metabolises blood

!lucose. &easures blood !lucose 0su!ar1 usin! a

micro-sensor and computes the insulin dose

required to metabolise the !lucose.



)nsulin pump or!anisation

enDisDoPo



)nsulin pump data-flow

InsulumInsuliPuolcomIreerecInsdelvercooll



ependability requirements

The system shall be available to deliver

insulin when required to do so.

The system shall perform reliability and

deliver the correct amount of insulin tocounteract the current level of blood su!ar.

The essential safety requirement is that

excessive doses of insulin should never bedelivered as this is potentially life

threatenin!.



ependability

The dependability of a system equates to itstrustworthiness.

" dependable system is a system that is

trusted by its users. 2rincipal dimensions of dependability are3

# "vailability%

# 4eliability%

# Safety%# Security



imensions of dependability

ava aeThe abto delire uesTheto des ecThtocatTtai



Other dependability properties

4epairability# 4eflects the extent to which the system can be repaired in

the event of a failure

&aintainability

# 4eflects the extent to which the system can be adapted tonew requirements%

Survivability# 4eflects the extent to which the system can deliver

services whilst under hostile attac%

rror tolerance# 4eflects the extent to which user input errors can be

avoided and tolerated.



&aintainability

" system attribute that is concerned with the ease ofrepairin! the system after a failure has beendiscovered or chan!in! the system to include newfeatures

5ery important for critical systems as faults are oftenintroduced into a system because of maintenanceproblems

&aintainability is distinct from other dimensions of

dependability because it is a static and not adynamic system attribute. ) do not cover it in thiscourse.



Survivability

The ability of a system to continue to deliver

its services to users in the face of deliberate

or accidental attac

This is an increasin!ly important attribute fordistributed systems whose security can be

compromised

Survivability subsumes the notion ofresilience - the ability of a system to continue

in operation in spite of component failures


18/45


ependability vs performance

+ntrustworthy systems may be rejected by their

users

System failure costs may be very hi!h

)t is very difficult to tune systems to mae them moredependable

)t may be possible to compensate for poor

performance

+ntrustworthy systems may cause loss of valuableinformation


19/45


ependability costs

ependability costs tend to increase exponentially

as increasin! levels of dependability are required

There are two reasons for this

# The use of more expensive development techniques andhardware that are required to achieve the hi!her levels of

dependability

# The increased testin! and system validation that is

required to convince the system client that the required

levels of dependability have been achieved


20/45


Costs of increasin! dependability

LowMHehD


21/45


ependability economics

'ecause of very hi!h costs of dependabilityachievement, it may be more cost effective toaccept untrustworthy systems and pay for

failure costs owever, this depends on social and political

factors. " reputation for products that can(tbe trusted may lose future business

epends on system type - for businesssystems in particular, modest levels ofdependability may be adequate


22/45



4eliability# The probability of failure-free system operation

over a specified time in a !iven environment fora !iven purpose

"vailability# The probability that a system, at a point in time,

will be operational and able to deliver therequested services

'oth of these attributes can be expressedquantitatively


23/45



)t is sometimes possible to subsume systemavailability under system reliability# Obviously if a system is unavailable it is not

deliverin! the specified system services owever, it is possible to have systems with

low reliability that must be available. So lon!as system failures can be repaired quicly

and do not dama!e data, low reliability maynot be a problem "vailability taes repair time into account


24/45


4eliability terminolo!y

Term Description

System failure An event that occurs at some point in time when

the system does not deliver a service as expected

by its users

System error An erroneous system state that can lead to system

behaviour that is unexpected by system users.

System fault A characteristic of a software system that can

lead to a system error. For example, failure to

initialise a variable could lead to that variable

having the wrong value when it is used.Human error or

mistake

Human behaviour that results in the introduction

of faults into a system.


25/45


$aults and failures

$ailures are a usually a result of system errors thatare derived from faults in the system

owever, faults do not necessarily result in systemerrors# The faulty system state may be transient and *corrected(

before an error arises

rrors do not necessarily lead to system failures# The error can be corrected by built-in error detection and

recovery

# The failure can be protected a!ainst by built-in protectionfacilities. These may, for example, protect systemresources from system errors


26/45


2erceptions of reliability

The formal definition of reliability does not alwaysreflect the user(s perception of a system(s reliability# The assumptions that are made about the environment

where a system will be used may be incorrect

# +sa!e of a system in an office environment is liely to bequite different from usa!e of the same system in a universityenvironment

# The consequences of system failures affects theperception of reliability# +nreliable windscreen wipers in a car may be irrelevant in a

dry climate# $ailures that have serious consequences 0such as an en!ine

breadown in a car1 are !iven !reater wei!ht by users thanfailures that are inconvenient


27/45


4eliability achievement

$ault avoidance# evelopment technique are used that either minimise the

possibility of mistaes or trap mistaes before they resultin the introduction of system faults

$ault detection and removal# 5erification and validation techniques that increase the

probability of detectin! and correctin! errors before thesystem !oes into service are used

$ault tolerance

# 4un-time techniques are used to ensure that systemfaults do not result in system errors and6or that systemerrors do not lead to system failures


28/45


4eliability modellin!

7ou can model a system as an input-outputmappin! where some inputs will result inerroneous outputs

The reliability of the system is the probabilitythat a particular input will lie in the set ofinputs that cause erroneous outputs

ifferent people will use the system in

different ways so this probability is not astatic system attribute but depends on thesystem(s environment


29/45


)nput6output mappin!

OeOut

Programo


30/45


4eliability perception

Us32


31/45


4eliability improvement

4emovin! 89 of the faults in a system will not

necessarily improve the reliability by 89. " study at

)'& showed that removin! :;9 of product defects

resulted in a


32/45


Safety

Safety is a property of a system that reflects the

system(s ability to operate, normally or abnormally,

without dan!er of causin! human injury or death and

without dama!e to the system(s environment

)t is increasin!ly important to consider software

safety as more and more devices incorporate

software-based control systems

Safety requirements are exclusive requirements i.e.

they exclude undesirable situations rather than

specify required system services


33/45


2rimary safety-critical systems

# mbedded software systems whose failure can cause the

associated hardware to fail and directly threaten people.

Secondary safety-critical systems

# Systems whose failure results in faults in other systems

which can threaten people

iscussion here focuses on primary safety-critical

systems

# Secondary safety-critical systems can only be consideredon a one-off basis

Safety criticality


34/45


Safety and reliability are related but distinct# )n !eneral, reliability and availability are

necessary but not sufficient conditions forsystem safety

4eliability is concerned with conformance toa !iven specification and delivery of service

Safety is concerned with ensurin! system

cannot cause dama!e irrespective ofwhetheror not it conforms to its specification

Safety and reliability


35/45


Specification errors

# )f the system specification is incorrect then the

system can behave as specified but still cause

an accident ardware failures !eneratin! spurious inputs

# ard to anticipate in the specification

Context-sensitive commands i.e. issuin! the

ri!ht command at the wron! time

# Often the result of operator error

+nsafe reliable systems


36/45


Safety terminolo!y

Term Definition

Accident (or

mishap)

An unplanned event or sequence of events which results in human death or injury,

damage to property or to the environment. A computer-controlled machine injuring its

operator is an example of an accident.

Hazard A condition with the potential for causing or contributing to an accident. A failure of

the sensor that detects an obstacle in front of a machine is an example of a hazard.

Damage A measure of the loss resulting from a mishap. Damage can range from many people

killed as a result of an accident to minor injury or property damage.

Hazard

severity

An assessment of the worst possible damage that could result from a particular

hazard. Hazard severity can range from catastrophic where many people are killed tominor where only minor damage results.

Hazard

probability

The probability of the events occurring which create a hazard. Probability values tend

to be arbitrary but range fromprobable (say 1/100 chance of a hazard occurring) toimplausible (no conceivable situations are likely where the hazard could occur).

Risk This is a measure of the probability that the system will cause an accident. The risk is

assessed by considering the hazard probability, the hazard severity and the probability

that a hazard will result in an accident.


37/45


Safety achievement

a=ard avoidance

# The system is desi!ned so that some classes of ha=ard

simply cannot arise.

a=ard detection and removal

# The system is desi!ned so that ha=ards are detected and

removed before they result in an accident

ama!e limitation

# The system includes protection features that minimise the

dama!e that may result from an accident


38/45


/ormal accidents

"ccidents in complex systems rarely have a sin!lecause as these systems are desi!ned to be resilientto a sin!le point of failure# esi!nin! systems so that a sin!le point of failure does

not cause an accident is a fundamental principle of safesystems desi!n

"lmost all accidents are a result of combinations ofmalfunctions

)t is probably the case that anticipatin! all problem

combinations, especially, in software controlledsystems is impossible so achievin! complete safetyis impossible


39/45


Security

The security of a system is a system propertythat reflects the system(s ability to protectitself from accidental or deliberate external

attac Security is becomin! increasin!ly important

as systems are networed so that externalaccess to the system throu!h the )nternet is

possible Security is an essential pre-requisite for

availability, reliability and safety


40/45


$undamental security

)f a system is a networed system and is

insecure then statements about its reliability

and its safety are unreliable

These statements depend on the executin!system and the developed system bein! the

same. owever, intrusion can chan!e the

executin! system and6or its data

Therefore, the reliability and safety

assurance is no lon!er valid


41/45


Security terminolo!y

Term Definition

Exposure Possible loss or harm in a computing system. This can be loss or

damage to data or can be a loss of time and effort if recovery is

necessary after a security breach.

Vulnerability A weakness in a computer-based system that may be exploited tocause loss or harm.

Attack An exploitation of a system vulnerability. Generally, this is from

outside the system and is a deliberate attempt to cause some damage.

Threats Circumstances that have potential to cause loss or harm. You can

think of these as a system vulnerability that is subjected to an attack.

Control A protective measure that reduces a system vulnerability. Encryptionwould be an example of a control that reduced a vulnerability of a

weak access control system.


42/45


ama!e from insecurity

enial of service

# The system is forced into a state where normal services

are unavailable or where service provision is si!nificantly

de!raded

Corruption of pro!rams or data# The pro!rams or data in the system may be modified in

an unauthorised way

isclosure of confidential information

# )nformation that is mana!ed by the system may beexposed to people who are not authorised to read or use

that information


43/45


Security assurance

5ulnerability avoidance# The system is desi!ned so that vulnerabilities do not occur.

$or example, if there is no external networ connection thenexternal attac is impossible

"ttac detection and elimination# The system is desi!ned so that attacs on vulnerabilitiesare detected and neutralised before they result in anexposure. $or example, virus checers find and removeviruses before they infect a system

xposure limitation# The system is desi!ned so that the adverse consequences

of a successful attac are minimised. $or example, abacup policy allows dama!ed information to be restored


44/45


>ey points

" critical system is a system where failure can lead tohi!h economic loss, physical dama!e or threats to life.

The dependability in a system reflects the user(s trustin that system

The availability of a system is the probability that it willbe available to deliver services when requested

The reliability of a system is the probability that systemservices will be delivered as specified

4eliability and availability are !enerally seen asnecessary but not sufficient conditions for safety andsecurity


45/45

>ey points

4eliability is related to the probability of an erroroccurrin! in operational use. " system with nownfaults may be reliable

Safety is a system attribute that reflects the system(s

ability to operate without threatenin! people or theenvironment

Security is a system attribute that reflects thesystem(s ability to protect itself from external attac

ependability improvement requires a socio-technical approach to desi!n where you consider thehumans as well as the hardware and software

critical system

Documents