safety analysis of software-intensive systems tor stålhane idi / ntnu
TRANSCRIPT
Safety Analysis of Software-intensive Systems
Tor Stålhane
IDI / NTNU
2
What is safetyA system is safe if it behaves in such a way
that it does not harms people, equipment or the environment.
Safety is a relationship between a system and its environment
Safety is not an add-on to a system but an integrated part that needs to be considered from day one of a development project.
3
What is safety analysis - 1
Safety analysis is the totality of activities that are used to identify
• Hazards that may rise when a system is put into operation.
• Ways to remove these hazards or reduce their consequences to an acceptable level.
• Actions needed throughout the system’s development to ensure that all safety requirements are implemented.
4
What is safety analysis - 2
The soft side of safety analysis: Collecting and analyzing info. The problems are human related.
• Collecting info from all stakeholders • Organize it in such a way that it can be
used to create– Safety requirements for development– Safety tests – Safety routines and procedures for the
operation and maintenance of the system
5
What is safety analysis - 3
The hard side of safety analysis: Defining barriers. The problems are related to both humans, software and hardware:
• How can we construct barriers against hazards in the software?
• How can we define operating procedures for handling crises?
6
Collecting info - 1
All stakeholders must be involved in the safety analysis since they all possess vital info.
Safety analysis is thus a people intensive process – critically dependent on
• The participants’ experience and knowledge.
• Our ability to elicit relevant info
7
Collecting info - 2
We need to identify
• All potentially dangerous events - hazards.
• The events’ consequences.
• The events’ probability or frequency – at least in qualitative terms.
• Important scenarios. The quality of the info from a person increases when the questions are related to a scenario.
8
Tools and methods - 1
The methods that we use in safety analysis – especially in the early phases – must be able to involve all stakeholders.
We need methods that are easy to
• Learn and understand
• Use on real-life problems
• Apply to software, hardware, people and routines and procedures.
9
Tools and methods - 2
Which tools and methods to use depend on who participate in the process, the info available and how it is represented.
The info available will depend on where in the development process we are.
The way the info is represented is, at least partly, something that we can influence. We have good experience with using UML diagrams in all phases.
10
Tools and methods - 3As we move from a concept to a high level
design and then on to detailed design and implementation, more and more
• Information will be available• Decisions will be made and thus leave us
with less freedom when making new decisions.
Thus, we will need different analysis methods in different phases of the system’s life cycle.
11
Project time and decisions
Time TD
Knowledge
Freedom of decisions
Experience
Concept HLD LLD Implementation
12
The concept phase
Most systems start as a concept, e.g.:
• Automatic shut-down of production when we discover a gas leakage.
• All patient info kept in a central database and be available for all that need it through a data network.
• Complete overview of all our trains – where they are, their speed and so on.
13
Electronic patient journal – Concept
Primary Physician
Nurse
Physician
Lab system
Patient journal system
Top level view – system and stakeholders
14
Experience Knowledge Experience Knowledge
Systemconcept
Tools and methods
Hazards andbarriers
Operationalenvironment
Stakeholder Stakeholder
15
Preliminary Hazard Analysis - 1
The preliminary hazard analysis is used early in the process. This is reflected in the level of details required in the PHA table.
We can include both hazards and the corresponding preventive actions – barriers.
Barrier descriptions are converted to system requirements.
16
Preliminary Hazard Analysis - 2
Hazard Cause Main effect Preventive action
Somebody retrieve wrong info
Wrong info inserted
Kill or hurt patient
Double check all patient info inserted
Stored info corrupted
Double store and check
Wrong patient id used at insertion or retrieval
Redundant patient info required for retrieval
: : : :
17
Requirements
Once we have decided to go ahead with the project, we need to elicit and document the requirements. These consist of two components:
• The functions used to fulfil the customer’s needs
• Barriers against hazards identified in the PHA
18
Use Case for Electronic patient journal
Nurse
Medication
Documents
Diagnosis
Orders and responses
Treatment plan
Physician
Primary Physician
Lab system
19
Needs Expectations
Customer Requirements
Hazards andbarriers
Systemconcept
Operationalenvironment
Methods and tools
New hazards and barriers
Experience Knowledge
Stakeholder
Experience Knowledge
Stakeholder
20
Safety in the requirements phase
Functional requirements – which services should the system offer to its users?
Use case diagrams and textual use cases have turned out to be two efficient ways of documenting this. They
• Are easy to understand for all stakeholders.
• Can be used as input to several safety analysis methods.
21
Use case for medication
22
Functional FMEA
Component Id Treatment plan
Function Failure mode
Local effect
Sys effect Actions Seriousness
Check current treatment
Return wrong data
Wrong decision
Patient can get hurt or killed
Check against other data available on this patient
H
Return data for another patient
H
Return no data
NoneSuspend decision
Alternative data source
L
Update treatment plan
Wrong update
Wrong data in journal
Wrong treatment
Implement update receipt
H
No update H
23
Misuse case Review
treatment plan
Review drug data
Review documents
Review diagnosis
Network is down
Wrong update
Delete data
Data islost
Unluckydoctor
Faulty system
Doctor<<threatens>>
<<threatens>>
24
High level design
When we enter high level design, all identified hazards and barriers have been converted to requirements.
The high level design can be documented for instance as
• Package diagrams
• High level class diagrams
• High level sequence diagrams
25
Part of electronic patient journal
Patient diagnosis
Patient drugdata
General patient info
Patient documents
Treatment plan
26
Experience Knowledge Experience Knowledge
Systemconcept
Tools and methods
New hazards
Operationalenvironment
Extendedrequirements
Barriers andtests
Stakeholders Stakeholders
27
Safety and design
Packages and classes can be viewed as components and we can thus make our safety analysis much more detailed.
Important methods that can be used at this stage are for instance:
• HazOp, for architectural design.
• Component FMEA
28
HazOp - 1
HazOp uses study nodes as units of investigation and guide words to help in the hazard identification process. This makes the method quite efficient for identifying hazards
On the other hand, HazOp also requires more information – the system’s architecture – to define the study nodes.
29
HazOp - 2
This is a simple version – more elaborate versions gives more info and requires more work.
Guideword Studynode
Consequences Causes Possiblesolutions
Less Generalpatientinfo
Incomplete infowhich can leadto wrongtreatment
Missing updates
Lost updates
Incompleteupdates
Check andsign-off forupdates
Mirrordatabase
30
Failure Mode Effect Analysis - 1
FMEA will systematically check each system component
• How can this component fail?
• What are the consequences for the component?
• What are the consequences for the system?
• How can we handle the hazard?
31
Failure Mode Effect Analysis - 2
Component Failure mode
Effect Handling or barrier
Seriousness
Patient drug data
Give wrong data
Wrong medication description. e.g. dosage
Check dosage against medication rules database.Prevent too high dosage
High
Incorrect or missing update
Outdated medication description, e.g. dosage
High
32
Failure Mode Effect Analysis - 3
The failure Mode Effect Analysis:• Offers a systematic walk-through of one or
more system components.• Focuses on preventions – barriers - rather
than cures and fixes.• Produces an easy-to-use list of hazards
and ideas on how they can be removed or handled.
33
Detailed design
Just as high level design, the detailed design can be documented for instance as packages, class diagrams and sequence diagrams.
We have more info than we had during high level design and we can thus make a more detailed safety analysis.
34
Patient info
Patient documents
Drug DBPatientdrug data
Treatmentplan
Test results
Currenttreatment
Drugdescription
If changes necessary
Update drug data
35
Experience Knowledge Experience Knowledge
Tools and methods
New hazards
Operationalenvironment
High level design
Barriers andtests
Barriers Detaileddesign
Stakeholder Stakeholder
36
Component FMEA
Component Id
Treatment plan
Failure mode Local effect Sys effect Actions Seriousness
Return wrongtreatment
Wrong info todoctor
Wrongdecision
Sanity check H
Update wrongdrug data
Wrong info inpatient’sjournal
Wrongmedication
Update receipt
H
Update drugdata forwrongpatient
H
: : : : :
37
Implementing barriers
All hazard analyses must lead to barriers that have one of the following effects:
• Prevent a hazard from leading to a problem.
• Prevent a problem from causing a dangerous event.
• Reduce the effect of a dangerous event if it cannot be prevented.
38
Bar
rier
1 Bar
rier
2 Bar
rier
3 Bar
rier
4 Bar
rier
5 Bar
rier
6
Risk Prob. Event
PreventionPrevent risk from becoming a problem
HandlingPrevent event from having bad consequences
Reduction Reduce effect of event
Barrier roles
39Risk RM
Minimum achievablerisk
RA
Acceptablerisk
Unmitigated riskfrom EUC
Ru
All barriers work as planned
Barr. 1Barr. 2Barr. 3Barr. n
Barrier failure
Barrier reliability
40
Realizing barriers
Barriers in software can be realized in several ways. It is important that they do not lead to a large increase in complexity.
One way to realize barriers is to use patterns such as:
• Façades or wrapper façades • Protected single channel• Sanity checks on values• Monitor - actuator
41
Façade pattern
42
Façade pattern sequence diagram
43
Wrapper façade pattern
44
45
Sanity check pattern
46
Monitor - actuator
47
Safety analysis research - 1
Research on safety analysis are concerned with some broad problem areas: How to
• Implement barriers to prevent or reduce the effect of dangerous events?
• Create safety analysis patterns?
• Elicit the necessary information from all stakeholders?
48
Safety analysis research - 2
Our current research in the area of software safety has focused on:
• Which methods are the easier to understand, learn and use?
• What is the relationship between method and system representation – is it e.g. easier to base an analysis on scenarios than on a requirements list?
49
Safety analysis research - 3
How can we
• Improve the safety analysis by making earlier experiences on similar systems available to all stakeholders?
• Most efficiently move from identified hazards to– Prevention, e.g. barriers– Tests – do the barriers work as intended?
50
Last but not least
It is possible to be too safe. A chainsaw with a fully protected blade is
• Absolutely safe• Absolutely useless
It is not possible to be absolutely safe. Whatever you do or don’t do, the probability of dying during the next hour is more than 10-6.
Make sure you have a nice day.