5. More on assessing performance Dr Mike Yearworth

Reader in Systems Engineering

©  University  of  Bristol  2014  

Notation

•  If  E  is  a  proposi8on,  an  interval  number  is  used  as  a  probability  measure  such  that  

P(E)=[Sn(E),  Sp(E)]  •  Where  Sn(E)  is  the  lower  bound  and  Sp(E)  the  upper  bound  of  probability  of  P(E)  e.g.  – P(E)=[0,0]  –  indicates  belief  that  E  is  certainly  false  – P(E)=[1,1]  –  indicates  belief  that  E  is  certainly  true  – P(E)=[0,1]  –  indicates  belief  that  E  is  unknown  

   

2

Interval Numbers Notation and the Italian Flag

•  P(E)=[1.0,1.0]      

•  P(E)=[0.0,0.0]  •  P(E)=[0.0,1.0]  •  P(E)=[0.4,0.9]  

3

Using ‘Hard Data’ - Mapping Value Functions

4

(Marashi  &  Davis  2006)  

Using ‘Hard Data’ - Mapping Value Functions

5

(Marashi  &  Davis  2006)  

Example of Mapping Likert Scales

6  

(Deakin  Crick,  Green,  Barr,  Shafi  &  Peng  2013)  

Summary of Judgements Required

•  Evidence  -­‐  for  and  against  separated  •  Sufficiency  -­‐  How  much  of  the  evidence  is  directly  relevant  to  

the  parent  process?    –  1  =  it  is  sufficient  on  its  own  to  fully  determine  the  success  of  the  

parent  •  Necessity  -­‐  Will  the  parent  fail  if  the  sub-­‐process  fails?  

–  1  =  it  is  certain  the  parent  process  will  fail  if  the  child  process  fails  –  Takes  over  if  evidence  against  is  large  

•  Dependency  -­‐  How  much  overlap  of  evidence  is  there  between  the  sub-­‐processes?  –  Issues  of  bias  and  redundancy  

7

How this works in a simple example

8  

Practical guide – heuristics

•  Experience  has  shown  that  es8ma8ng  necessity,  sufficiency,  and  dependency  condi8ons  is  hard  

•  In  prac8ce  PeriMeta  is  used  in  in  a  way  that  assumes  that  the  number  of  processes  per  level  is  >2  and  usually  ~5±2  –  Using  default  parameters  of  Necessity=0.3,  Sufficiency=0.4,  Dependency=0  (independent)  

•  Note  that  the  point  of  using  HPM  is  problem  structuring  with  the  aim  of  interven8on,  not  predic8on  

9

Review PeriMeta model

•  In  groups…  •  If  model  is  not  already  in  PeriMeta,  now  is  the  8me…  •  Review  model  and  check  necessity  and  sufficiency  condi8ons  –  Default  can  be  used  if  the  number  of  dependent  processes  is  ~5  

–  If  <5  review  the  necessity  and  sufficiency  links  (check  with  Mike/Dan)  

•  Does  this  change  anything  in  terms  of  deciding  where  to  place  focus  for  interven8on?  

10  

References Blockley,  D.,  &  Godfrey,  P.  S.  (2000).  Doing  it  differently  :  systems  for  rethinking  construc7on.  

London:  Thomas  Telford.  Davis,  J.  P.,  MacDonald,  A.  D.  J.,  &  Marashi,  S.  E.  (2007).  Integrated  performance  measurement  to  

support  strategic  decision  making  in  engineering  organisa7ons.  Paper  presented  at  the  IEEE  Interna8onal  Conference  on  Industrial  Engineering  and  Engineering  Management,  Singapore.    

Deakin  Crick,  R.,  Green,  H.,  Barr,  S.,  Shafi,  A.,  &  Peng,  W-­‐J.  (2013).  Evalua7ng  the  wider  outcomes  of  schooling:  the  Oasis  ECHO  project  (pp.  212).  Bristol:  University  of  Bristol.  

Hall,  J.  W.,  Blockley,  D.  I.,  &  Davis,  J.  P.  (1998).  Uncertain  inference  using  interval  probability  theory.  Interna7onal  Journal  of  Approximate  Reasoning,  19(3-­‐4),  247-­‐264.    

Marashi,  E.,  &  Davis,  J.  P.  (2006).  An  argumenta8on-­‐based  method  for  managing  complex  issues  in  design  of  infrastructural  systems.  Reliability  Engineering  &  System  Safety,  91(12),  1535-­‐1545.  doi:  10.1016/j.ress.2006.01.013  

Marashi,  S.  E.,  Davis,  J.  P.,  &  Hall,  J.  W.  (2008).  Combina8on  methods  and  conflict  handling  in  eviden8al  theories.  Interna7onal  Journal  of  Uncertainty  Fuzziness  and  Knowledge-­‐Based  Systems,  16(3),  337-­‐369.  doi:  10.1142/s0218488508005315  

11