lian duan and mats p. e. heimdahlfm.csl.sri.com › verisure2015 › talks › verisure beta.pdfdr....

VeriSure'15 7/18/2015

Dr. Mats Heimdahl 1

Software Engineering Center


Representing Confidence in

Assurance Case Evidence

Lian Duan and Mats P. E. Heimdahl University of Minnesota Software Engineering Center

Department of Computer Science and Engineering

University of Minnesota

4-192 EE/CS; 200 Union Street SE

Minneapolis, MN 55455

VeriSure'15 1 7/18/2015

Funded by CNS-0931931

and CNS-1035715


Assurance Case Example

2 7/18/2015 VeriSure'15

Abstractions OK?

Environment OK?

Tool OK?

Fraud?

Oracles OK?

Test harness OK?

Test vs Production?

Fraud?

VeriSure'15 7/18/2015

Dr. Mats Heimdahl 2


Confidence

• Need to have confidence in the claims made in the assurance case

– Does not matter what approach to the case we take

• Approving a system is always a judgement call

• Can model it qualitatively or quantitatively

– Qualitatively: • Separate confidence case, acceptance criteria, …

– Quantitatively: • Use a single number, a range, or a distribution

7/18/2015 VeriSure'15 3


Previous Work: Bayesian Networks

• Evidence nodes (quantitative information)

• Links (qualitative information)

7/18/2015 VeriSure'15 4

VeriSure'15 7/18/2015

Dr. Mats Heimdahl 3


Previous work:

Dempster-Shafer Theory

• Separate out uncertainty

• Belief, Disbelief, Uncertainty

7/18/2015 VeriSure'15 5


Jøsang’s Opinion Triangle

• Standard logic: bottom part (disbelief to belief)

of triangle

• Top vertex: uncertainty

• An opinion is

{belief, disbelief, uncertainty}

• Base rate: fourth variable

(prior belief)

7/18/2015 VeriSure'15 6

VeriSure'15 7/18/2015

Dr. Mats Heimdahl 4


Our Proposal

• Use the beta distribution to represent confidence

• Mapping to Jøsang’s opinion triangle

7/18/2015 VeriSure'15 7


Beta Distribution

• Continuous version of

binomial distribution

• Finite range

• Versatile

• 2nd order distribution:

probability of

probabilities

7/18/2015 VeriSure'15 8

VeriSure'15 7/18/2015

Dr. Mats Heimdahl 5


Uncertainty vs. Split Opinion

• {0.0, 0.0, 1.0} – Full uncertainty

• {0.5, 0.5, 0.0} – Split opinion

7/18/2015 VeriSure'15 9


More Beta/Opinion Triangle:

7/18/2015 VeriSure'15 10

VeriSure'15 7/18/2015

Dr. Mats Heimdahl 6



7/18/2015 VeriSure'15 11



7/18/2015 VeriSure'15 12

• Visualize as spikes at specific belief values

0 1 .5

VeriSure'15 7/18/2015

Dr. Mats Heimdahl 7



7/18/2015 VeriSure'15 13


Testing Evidence

7/18/2015 VeriSure'15 14

VeriSure'15 7/18/2015

Dr. Mats Heimdahl 8


Logical Argument Example

7/18/2015 VeriSure'15 15


Software Node: Ev1 Ev2

• Consensus operator:

7/18/2015 VeriSure'15 16

VeriSure'15 7/18/2015

Dr. Mats Heimdahl 9


Hardware Node: Ev3 Ev4

• Consensus operator

7/18/2015 VeriSure'15 17


No Over-Radiation

• Logical OR

7/18/2015 VeriSure'15 18

VeriSure'15 7/18/2015

Dr. Mats Heimdahl 10


Work in Progress

• Propose the use of the Beta distribution, and its duality with the opinion triangle/subjective logic, to represent confidence in assurance cases

• Work in progress

– Combination of distributions in an assurance case

– Weighing evidence

– Elicitation of confidence estimates

– Sensitivity analysis

7/18/2015 VeriSure'15 19


Summary

7/18/2015 VeriSure'15 20

Thank You

VeriSure'15 7/18/2015

Dr. Mats Heimdahl 11


Summary

7/18/2015 VeriSure'15 21

Thank You

Questions: • Should we quantify confidence/trust?

• Can we elicit belief in any reliable manner?

• Is Beta the answer?

• Can we have “belief templates” for various types

of evidence?

• Will this ever be better than educated guesses?

lian duan and mats p. e. heimdahlfm.csl.sri.com › verisure2015 › talks › verisure beta.pdfdr....

Documents