system safety - m12 safety cases and arguments v1 · m12 safety cases and arguments v1.4 matthew...

System SafetyM12 Safety Cases and Arguments V1.4

Matthew Squair

UNSW@Canberra

15 June 2016

1 Matthew Squair M12 Safety Cases and Arguments V1.4

1 Introduction

2 Overview

3 Methodology

4 But do safety cases work?

5 Limitations, advantages and disadvantages

6 Conclusions

7 Further reading


Introduction

1 Introduction

2 Overview

3 Methodology



6 Conclusions

7 Further reading


Introduction

Learning outcomes

Understand what a safety case is

Be able to critically review the content and argument of a safety case

Be able to structure and prepare the content of a safety case

Understand the strengths and weaknesses of the technique


Overview

1 Introduction

2 Overview

3 Methodology



6 Conclusions

7 Further reading


Overview

Overview

“The Nimrod safety case process was fatally undermined by a generalmalaise: a widespread assumption... that the Nimrod was ’safeanyway’ (because it had successfully flow for 30 years) and the task ofdrawing up the safety case became essentially a paperwork and’tickbox’ exercise.”

— C. Haddon Cave, The Nimrod Review


Overview

Overview

Safety cases

Originated in the British chemical industry CIMAH regulations

Applied to oil industry after the Piper Alpha oil rig fire

Applied to UK Rail after Clapham junction accident

Have become part of the EU safety culture

Embedded in various safety standards

DEF-STAN 00-56DEF (AUST) 5679Australian DMO SAMS FrameworkCMMI SAFE+IEC 61508


Overview

Overview

Despite it’s prevalence there are serious concerns about it’s practicalapplication [Haddon-Cave 2009] and theoretical underpinnings

We’ll look at the theory and application of safety cases with a focus onarguments in the context of acquisition

We’ll also discuss the problems and limitations of safety cases

Safety cases embody argumentation, logic, and epistemology. Humanbeings have been wrestling with these concepts since the dawn ofcivilization


Overview

How is a safety case different to MIL-STD-882?

A MIL-STD-882 system safety program:

Is acquisition focused (customer-supplier)

Addresses proximal (system) causes of accidents

Safety Assessment Report is analogue ’ish’ to a safety case

However, a safety case:

Can be operation (operator-regulator)

Convince a regulator the plant is safe to operate (WHS)

Can be acquisition developed (DEF STAN 00-56)

Can be goal (more usual) or rule/standard based:

Safety cases have traditionally formed part of goal (performance) basedsafety regimesProvide the strategy of how performance goals are translated intosolutions


Overview

Why do it?

Various reasons

You may need a tool to manage operational safety

You may wish to reduce liability risk

The regulator may require as a ‘permit to operate’

You may want to structure and organise safety documentation

You may want to communicate system risk to stakeholders

Safety cases are inherently technical and political documents

Be clear about the purpose

Different stakeholders may mean very different things when it comes tosafety cases, be clear about your purpose and who it serves when youprepare one


Overview

Key definitions

Safety argument. A safety argument is a clear, comprehensive anddefensible argument that explains how the available evidence supports theoverall claim of acceptable safety within a particular context [Kelly 1998]

Safety case. A safety case is a structured argument, supported by a bodyof evidence, that provides a compelling, comprehensible and valid casethat a system is acceptably safe for a given application in a givenenvironment (i.e a context) [MOD (UK) 2007]

Safety case report. The physical artifact(s) that presents the safetyargument and case. Normally the safety case report is not a standalonedocument and will refer out to supporting evidence.


Methodology

1 Introduction

2 Overview

3 Methodology



6 Conclusions

7 Further reading


Methodology

Methodology [Bishop, Bloomfield 1998]

1 Identify safety requirements

2 Identify system architecture and outline the safety case

3 Assessment (preliminary) of concept design safety trades

4 Progressive elaboration of the design & safety case in parallel

5 Integrate into final safety case

6 Plan for long-term support infrastructure

7 Review and approval8 Long-term monitoring and audits

of areas of concernof support processesto gather field evidence to support assumptions

9 Revise to reflect system and context changes


Methodology Contents of a safety case

Contents

Contains at a minimum[Kelly 1998]:

Supporting evidence on which the case is based, because argumentwithout evidence is unfounded

A high level argument, because evidence without argument isunexplained

May include a number of separate sub-argumentsA convergent conclusion as to the acceptability of the system

A meta-argument as to why the argument and evidence should bebelieved because both evidence and argument can be faulty[Hawkins et al., 2011]

Is the totality of the safety evidence NOT just a safety case report

Structure and organisation is essential to achieve clarity



Safety case report

The exact content may be defined by a regulator, but generally:

scope

system description

system hazards

safety requirements

risk assessment

hazard control / risk reduction measures

safety analysis / test

safety management system

development process justification

conclusions



Toulmin’s model of practical arguments

Current practices in formal safety argument are based on the practicalargument model [Toulmin 1958]. Focuses on the justification aspects ofarguments rather than inferential. Argument parts consist of facts(evidence), conclusions(claims), warrants, backing and qualifiers

The claim This is the expressed opinion or conclusion that the arguerwants accepted by the audience.

The warrant is why it’s considered to move from the fact to the conclusion

The rebuttal is a legitimate constraint that may be placed on theconclusion drawn

Backing is evidence introduced if the warrant is prima facie not credible

Qualifier is an adverbial phrase indicating the strength of claim (e.g.,certainly, presumably, probably, possibly, suggests, implies etc)



Toulmin’s model (cont’d)



A small philosophical quibble

The problem is that Toulmin developed his model so that one couldanalyse an argument, that is the word argument is used in the verb sense

Safety arguments tend to inherently skew to an advocacy position, and therebuttal part of Toulmin’s model gets overlooked, that is in safetyarguments the word argument is used as a noun

From there it is a small step to the narrative fallacy e.g. presenting all thatgood data that the system is safe

Of course there’s very little evidence of rare catastrophic events becausethey’re, well, rare...


Methodology Formal notations

The problem with words

For larger safety arguments, there is the risk that the amount of words willobscure the argument. One solution is to use ‘semi-structured prose’,where standard terms (evidence, claim, strategy, justification etc) arehighlighted

Example

SFAIRP satisfied argument. The argument establishes the claim (c1),that the system design satisfies the so far as is reasonably practicablecriteria in the context of a definition of what is constitutes reasonablypracticability. To establish the top claim, two sub-claims (c1, c2) areestablished: (c1) all identified hazards have been eliminated, or their riskreduced as low as is reasonably practicable and (c2) that the residual riskis not unacceptably high. A Backing argument is provided thatsupports....



The problem with words (Pt II)

Other techniques can be applied [Holloway, 2008]:

Use formatting of paragraphs, indenting and numbering.

Mathematical proof (given, by layout) format supported by tabularstatement/reason(s) pairs, John Rushby takes this further

LISP programming language format

We can also augment the text, or supplant it all together, with a graphicaldepictions of the argument



Graphical notations for safety arguments

Two formal graphical notations are available:

Goal Structuring Notation (GSN). Developed by Kelly & others,there is a GSN community standard

Claims, Arguments, Evidence (CAE). Developed by Bishop &others, supported by Adelard’s Safety Case Editor tool

Both are graphical in nature to assist in clarity of argument

Both are based on Toulmin’s practical argument structure

Both implictly allow inductive (implies but not entails) style argument

Claim = Goal, Strategy = Argument, Solution = Evidence

Clarity does not denote soundness

The use of one particular notation or another does not infer any greater orlesser soundness upon the actual worth of the argument



Graphical notations for safety arguments

GNS versus CAE notation


Methodology Developing the safety case

Developing the safety argument (GSN notation)

1 Establish top level goals (customer/statutory)

2 Record the stakeholders for the goals

3 Define derived requirements (standards, codes etc)

4 Establish (3) as goals (or constraints) and link to top goals

5 Break down the top level goals into sub-goals

6 Show how design & analysis decisions meet goals via strategies

7 Record the decisions as they are made

8 Justify strategies

Evidence versus argument

Evidence without argument is unexplained, argument without evidence isunfounded



How convincing is the argument?

Or how do we know that a safety argument is sound?

John Rushby [Rushby 2016] recommends a separation of concerns so thateach argument step is supported either by:

sub-claims: an interior or reasoning step, or

evidence: a leaf or evidential step, but

not by a mixture (but we can allow evidential steps in whichsub-claims are used as assumptions)

Argument can be sound but weaker or stronger given whatever thresholdwe set for the evidence (e.g. formal proof versus white-box testing, versusblack box testing)

Changing the argument can affect it’s strength, see for example DO-178Cassurance level objectives



Argument soundness

There are two kinds of argument step, interpreted differently:

Evidential argument steps connect the real empirical world to theconceptual world of claims (epistemology)

Interior argument steps are about reasoning (logic)

Keep the interior argument as deductive logic so we can test it,inductive uncertainty is pushed to the leaf or evidential argumentsteps

Argument is sound if:

Interior reasoning is deductively valid (test logic)

inductive reasoning is weakall inductive uncertainty should be located in the evidential stepsany remaining interior step that is still inductive must be flagged usinga qualifier (‘implies’, ‘suggest’)

Evidence crosses some threshold of trust



Argument strength

We can strengthen any inductive parts of an argument through addingsupporting confidence claims that increase our confidence in the argument(artefact or process quality claims are examples)

Confidence claims are predominantly associated with the evidential claimpart of the argument

We can use probability measures to express degree of belief in the evidence

We can reduce or increase threshold for evidence to strengthen or weakenthe argument

Arguments can also be weakened or strengthened by ‘pruning’, see forexample software assurance levels such as DO-178’s DAL objectives



Example fragment of a safety argument in GSN notation



Dealing with size and complexity

For large and complex systems the associated safety case can also becomequite large and complex. How do we deal with this?

Utilise safety case modules to hide detail

Modules interact through defined interfaces

Utilise safety case patterns to standardise credible arguments



Example modular safety case

Figure: Eurocontrol RVSM pre-implementation safety case



Example modular safety case (cont’d)

Figure: Eurocontrol RVSM Implementation module30 Matthew Squair M12 Safety Cases and Arguments V1.4


Safety case patterns

Figure: Safety pattern: functional safety argument


Methodology Maintaining the safety case

Safety case maintenance

In theory, a safety case should be maintained till system retirement

Example

The Long Term Safety Review of the U.Ks Magnox reactors, quoted in[Kelly 1998] found that lack of maintenance to the original safety case had causedit to become inconsistent with current plant design and operations. The reviewfurther found that adding to and re-evaluating a safety case that has become outof date with respect to current safety standards was problematic

In practice, unless effort is expended to maintain the case it rapidly fallsout of date

A commitment to maintain requires regulatory & corporate buy in

For some facilities (such as nuclear) the system life may be up to acentury, longevity of evidence becomes a problem


Methodology Maintaining the safety case

Safety case maintenance

One of the biggest challenges is maintaining the safety case in the face ofsystem changes

We would like to use the safety case to assess changes for safety impact

We also have to repair the case after a change has been made, hopefully ina cost effective fashion

A graphical safety argument with traceability structures is invaluable forthese purposes [Kelly, McDermid 2001]


Methodology Challenging the safety case

Safety arguments as scientific hypothesis

The best tool that we have for differentiating between a good theory and abad one is the scientific method:

our hypothesis is that our system is safe

the argument is why we think this is justified

in science a justifiable hypothesis is not considered proven

in science the hypothesis is then challenged by others

but with safety argument is this (ever) the case?

The safety case as ’proof’ fallacy

An unchallenged safety case is essentially an appeal to authority argument,authority in this case being how impressive the report is



So how do we challenge a safety case?

Four broad avenues of attack:

Deconstruction

Refutation

Disconfirming evidence

And...

proof by construction, otherwise known as having an accident ornear miss

The above might seem a lot but (for example) a claim that the likelihoodof a LOCA accident is 10−9 per reactor year is a very strong statement,and strong statements demand strong proof surely?



So how do we challenge a safety case?

Four broad avenues of attack:

Deconstruction

Refutation


And... proof by construction, otherwise known as having an accident ornear miss

The above might seem a lot but (for example) a claim that the likelihoodof a LOCA accident is 10−9 per reactor year is a very strong statement,and strong statements demand strong proof surely?



Deconstruction

Based on the work of french philosopher Jacque Derrida on the theory ofmeaning (and it’s inherent indeterminacy) and his use of it in critiquingphilosophical arguments [Armstrong, Paynter 2002]

Derrida’s view on arguments

An argument is defined by what it ignores and the perspectives it opposes(explicitly or implicitly)



Deconstructionist technique

Develop a counter argument that seems warrantable and use this toexpose the internal flaws and contradictions in the original case

1 Reversal. Reverse the argument, ignore how warranted the original is& look for warrantable counter-arguments

2 Displacement. Compare the relative warrantedness of both3 Evaluate the three possible end states

The original argument is found to need revisionThe counter argument is found to need revisionThey both turn out to be equally compelling1

Apply this to the higher level claims (strategies) of the safety argument

The results of this deconstruction can be used as evidential steps in thebacking argument for the safety case

1Due to the limits of deductive closure37 Matthew Squair M12 Safety Cases and Arguments V1.4


Deconstruction (Class exercise)

Modelling software reliability

Argument. Software failures occur randomly because of the random nature ofinputs from the environment that trigger latent faults and that we can applyclassical reliability techniques.

What might be a warrantable counter argument, or arguments?



Refutation of argument [Greenwell et al. 2006]

Challenge the specific arguments on the basic of fallacious argumentstructures and refute them

Apply this to the interior deductive logic of the safety argument




Challenge the evidence with disconfirming evidence

Based on Karl Popper’s concept of the science project as one of trying todisconfirm theories not confirm them

Consider

Quality of the evidence provided (pool size, outlier handling, magicbullet approaches)

Hazard control coverage metrics (is the argument vulnerable)

Independence and dissimilarity of evidence sources

Then go out and gather strongly disconfirming evidence that targets thegaps

Apply this to the evidential steps in the safety argument


But do safety cases work?

1 Introduction

2 Overview

3 Methodology



6 Conclusions

7 Further reading


But do safety cases work?

Practical and theoretical problems with the approach

A number of of significant safety cases have been reviewed, and problemsfound with them

Magnox reactor safety review

Haddon enquiry into the Nimrod disaster

Ladkin analysis of the EUROCONTROL RVSM safety case

Knight analysis of Opalinus Clay Nuclear repository safety case

None of these were minor projects, so it appears that even when great careshould be taken, flawed arguments still appear

The theoretical problem is that for high consequence systems thelikelihood must be very, very low and we must have a very high faith in theargument that this is so. Do we?


Limitations, advantages and disadvantages

1 Introduction

2 Overview

3 Methodology



6 Conclusions

7 Further reading



Limitations of the method

Limitations

Relies upon correspondence between safety argument and safety case

Relies upon peoples ability to reason and argue effectively, there’s nota lot of evidence that people are actually good at this



Advantages

Advantages are that

Is almost mandatory if working in a goal based regulatory environment

Is invaluable in organising the safety program documentation ’tail’

Can promote thought and discussion, if used appropriately

Can provide a change safety impact assessment capability in service



Disadvantages

Disadvantages are that it

Can become over time, another tick the box exercise

Is vulnerable to the narrative fallacy

Has a tendency to become an advocacy piece

Is very hard to review effectively without formal training

Can become an administrative burden that is perpetually chasing thesystem


Conclusions

1 Introduction

2 Overview

3 Methodology



6 Conclusions

7 Further reading


Conclusions

Conclusions

Safety cases emerged out of the political and industrial landscape ofEngland in the late 1970’s, they reflect a particular societal viewpoint onboth who should be responsible for managing major hazards should bemanaged and therefore how they should manage them.

They are in the end another tool, neither an end in themselves nordemonstrably the only way to assure the safety of complex systems.

Their current demonstrated deficiencies perhaps more demonstrate thedifficulty humans have in arguing rigorously and logically, than any specificlimitations of the method


Further reading

Bibliography

[Armstrong, Paynter 2002] Armstrong, J. M. and Paynter, S. P. (2002). Safe Systems:Construction, Destruction and Deconstruction. In: Redmill, F. and Anderson, T.(eds.), Current Issues In Safety Critical Systems, pp. 63-76, Springer-Verlag, Berlin.

[Bishop, Bloomfield 1998] Bishop, P. G. & Bloomfield, R. E. (1998). A Methodologyfor Safety Case Development. In: F. Redmill & T. Anderson (Eds.), IndustrialPerspectives of Safety-critical Systems: Proceedings of the Sixth Safety-criticalSystems Symposium, Birmingham 1998.

[DoD (US) 1993] DoD (US) (1993) Standard Practice for System Safety (1993) USDept of Defense Standard MIL-STD-882C, 19 January 1993.

[Greenwell et al. 2006] Greenwell, W. S, Holloway, M., C. Knight, J.C., (2006) ATaxonomy of Fallacies in System Safety Arguments, Proceedings of the 2006International System Safety Conference.

[Haddon-Cave 2009] Cave, C.H. (2006) An Independent Review Into the Broader IssuesSurrounding the Loss Of The RAF Nimrod MR2 Aircraft XV230 In Afghanistan in2006, The Stationary Office, Tech. Rep., 2006


Further reading

[Hawkins et al., 2011] Hawkins, R., Kelly, T., Knight, J. and Graydon, P. (2011) Anewapproach to creating clear safety arguments, in Proc. SafetyCritical SystemsSymp., Feb. 2011.

[Holloway, 2008] Safety case notations: Alternatives for the non-graphically inclined?, In3rd IET International Conference on System Safety, The Institutions of Engineeringand Technology, Birmingham, UK, Oct. 2008.

[Kelly, McDermid 1997] Kelly T, McDermid J. (1997) Safety case construction andreuse using patterns. In: Proc. 16th Intl. Conf. Computer Safety, Reliability, andSecurity (SAFECOMP97). New York, 1997.

[Kelly 1998] Kelly, T.P., (1998) Arguing Safety, A Systematic Approach to ManagingSafety Cases, Doctoral Thesis, Dept of Computer Science, University of York 1998.

[Kelly, McDermid 2001] Kelly T, McDermid J. (2001) A systematic approach to safetycase maintenance. Reliability Engineering and System Safety 2001;71(3):271-284.

[MOD (UK) 2007] UK MoD (2007) Defence Standard 00-56 Issue 4: Safetymanagement requirements for defence systems, HMSO.


Further reading

[Rushby 2016] Rushby, J. (2016) On the Interpretation Of Assurance Case Arguments.In: Proc. of the Second International Workshop on Argument for Agreement andAssurance (AAA 2015), Keio University, Kanagawa, Japan, November 2015.

[Toulmin 1958] S. E. Toulmin, S.E., (1958) The Uses of Argument, CambridgeUniversity Press, 1958.


system safety - m12 safety cases and arguments v1 · m12 safety cases and arguments v1.4 matthew...

Documents