risk engine mark 2: technical aspects and...

___________________________________________________________________________

2012/SCSC/WKSP/005

Risk Engine Mark 2: Technical Aspects and Application

Submitted by: RMIT University

Workshop on Developing a Harmonised Electrical Equipment Regulatory Risk

Assessment ToolSingapore

15-16 May 2012

5/15/2012

1

Risk Engine Mark 2: Technical Aspects and Application

Dr Adrian SchembriSenior Lecturer (Stats & Industry)

Assoc Prof. Anthony BedfordDeputy Head of School

School of Mathematical and Geospatial SciencesRMIT University

May, 2012

Overview of the Presentation

Discuss the Broad Aims of the Project

Project Team and Acknowledgements;

Background to the Risk Engine;

Discussion on the Application of Risk Engine Mark 2.

RMIT University©2011 2Risk Engine Validation

5/15/2012

2

Aims of the Project

Review the technical aspects of the Risk Engine proposed for implementation in Australia;

Assemble a panel of experts with a strong knowledge of OH&S and electrical engineering to review the Risk Engine;

Critique the statistical aspects of the engine, and utilise advanced statistical techniques to validate the model;

Present the strengths and areas in need of further development i t th Ri k E i b i d t d i A t li

RMIT University©2011

prior to the Risk Engine being adopted in Australia;

Provide a series of recommendations and possible further developments to enhance the engine;

Development of an application for Risk Engine Mark 2.

3Risk Engine Validation

Project Team and


Project Team and Acknowledgements


5/15/2012

3

RMIT University: Project Team

School of Mathematical and Geospatial Sciences

> Project Team > Background > Stats Analysis > Risk Index > Expert Analysis> Recommendations

Dr. Adrian Schembri

Assoc. Prof. Anthony Bedford

Mr. Jonathan Sargent

School of Applied Sciences


Assoc. Prof. Susanne Tepe

Mr. Leo Ruschena


Other Collaborators: Mick Logan

ERAC Secretariat El t i l S f t Offi Q l d


Electrical Safety Office Queensland

Peter Lamont ERAC Coordinating Chair

ERAC Committee members

Peter Morfee


Principal Technical Advisor Energy Safety; Ministry of Economic Development

Vallabh Patel Statistical Analyst


5/15/2012

4

Background to


Background to the Risk Engine


Rationale for Developing a Risk Engine

Enables a risk assessment to be conducted on a range of


gproducts, Electrical and Electronic Equipment (EEE), using regulatory and non-regulatory controls;

Provides a means of attributing a level of regulatory intervention required for electrical products;

In this regard, low, medium and high risk classifications can be ll t d t d t i tit ti th d l


allocated to products using a quantitative methodology;

An advantage of quantitative analysis is that it does not rely on continual input from industry experts, thus a quantitative risk engine is resource efficient.


5/15/2012

5

Introduction to P Factors

P refers to the Probability Factor


P is the sum of a set of 10 individual factors that contribute to [the potential for the equipment to create a hazard] the likelihood that a class of equipment will be non-compliant and thus create a hazard (Morfee, 2009);

Examples: Product is not regulated in Australia;


g ; Product subject to a deviation from the relevant

international Standard; Product not generally regulated in Asia; There is significant compliance cost disincentive.


Introduction to T Factors

Also referred to as the Consequences Factor, the T Factor considers the technical features the product has that make it


considers the technical features the product has that make it likely to create harm or damage if not compliant with its applicable product standard;

The T Factor is currently the sum of a set of 20 individual factors;

Examples:


Product that is likely to be moved during or between uses; Product that relies on guards and barriers to prevent

mechanical injury; Product is likely to be used by unsupervised children; Product that provides an electrical safety function.


5/15/2012

6

Characteristics of the Risk Engine


The sum of individual T factors and sum of individual P factors is plotted on a graph;

Each of the individual P and T factors is given an equal weighting.


Plotting the ‘T’ and ‘P’ Factors

16

18


0

2

4

6

8

10

12

14

Pro

bab

ilit

y f

acto

r -

P


0 5 10 15 20 25

Technical factor - T

For this product, 1 P factor and 4 T factors are present.

For this product, 3 P factors and 11 T factors are present.


5/15/2012

7

The Delineation Lines High Risk ZoneRegulatory Control Required.


4

6

8

10

12

14

16

18P

rob

abil

ity

fac

tor

- P


0

2

0 5 10 15 20 25Technical factor - T

Low Risk Zone Medium Risk Zone


Classification of High Risk Products On the Declared Articles List in New Zealand

18


2

4

6

8

10

12

14

16

Pro

bab

ilit

y f

acto

r -

P

The majority of products considered to be high risk were classified in the high risk zone.


0

0 5 10 15 20 25Technical factor - T

Several products were placed marginally in the medium risk zone.


5/15/2012

8

Statistical Analysis


yof the Risk Engine


Aims of the Statistical Analysis

Examine the level of independence or interdependence between the individual T factors and the individual P factors;


between the individual T factors and the individual P factors;

Examine the relationship between T and P factors, and whether certain T and P factors are correlated;

Examine the ability of R, which is equal to T x P to assign a level of risk to products;

E l h i f i f d l


Evaluate the consistency of expert ratings of products as low, medium and high risk.

Identify whether experts ratings differ significantly from classifications based on the Risk Engine.


5/15/2012

9

Stepwise regression analysis used to determine whether

Aims of the Statistical Analysis


p g ycertain T and P factors accounted for a significant proportion of variability in Risk (T x P) scores;

Cluster analysis used to determine those products that shared similar T and P factors;

Development of two independent ratings systems as a grounds f i ith th t Ri k E i


for comparison with the current Risk Engine.


Independence of ‘T’ and ‘P’ Factors A key objective was to establish levels of independence

between the variables that determine the level of risk for a


given product;

In linear predictive modelling, independence between the inputs is a key assumption; violation of this assumption is termed multicollinearity and reduces the predictive power of the model as statistically significant variables can be forced to become insignificant by variables that share dependence;


Theoretically, if certain T or P factors were not independent, we would expect that when one T factor was present, other T (or P) factors would also be present, thus the correlation would be high.


5/15/2012

10

Independence of ‘T’ and ‘P’ Factors

This analysis followed a three-stage correlational analysis:


Correlation between each T factor with every other T factor;

Correlation between each P factor with every other P factor;

Correlation between each of the T and P factors


Correlation between each of the T and P factors.


Independence of ‘T’ Factors

The total sample of T factors incorporated 20 statements;


When all T factors are correlated with all other T factors, a total of 190 correlations are generated;

Of these, 20 were found to be significant, which is equivalent to 10.5% of all possible correlations between T factors;

In total, half of all T factors were correlated with one or more


other T factors, whilst half of the T factors were not significantly correlated with any other T factors.


5/15/2012

11

Independence of ‘P’ Factors The total sample of P factors incorporated 10 statements;

When all P factors are correlated with all other P factors, a total


of 45 correlations are generated. Of these, 11 were found to be significant, which is equivalent to 24.4% of all possible correlations between P factors;

In total, 8 P factors were correlated with one or more other P factors, whilst 2 P factors were not significantly correlated with any other P factors;


The two P factors found to not correlate with any others included: The dominant supplier's marketplace applies a standard

considered to be inadequate for New Zealand; Compliance with the applicable Standard is difficult to

achieve.


Relationship between T and P Factors

When all T factors are correlated with all P factors, a total of


,200 correlations are generated;

Of these, 26 were found to be significant, which is equivalent to 13% of all possible correlations between the T and P factors;

In total, 14 of the 20 T factors were found to significantly correlate with one or more P factors, whilst all 10 P factors

i ifi tl l t d ith t l t T f t


were significantly correlated with at least one T factor.


5/15/2012

12

Independence of ‘P’ Factors


The strongest correlations were found between the following P factors:

Product generally incorporates new Technology and Product is not regulated in Australia;

Product subject to a deviation from the relevant international Standard and The applicable international Standard is considered to be inadequate.


Correlation between Expert Ratings and T and P Factor Scores


Correlations between expert majority ratings (low, medium or high) and the cumulative sum of T and P factors were conducted.

Correlated Variables Correlation Coefficient (p-value)

E t M j it R ti d 313 ( 005)


Expert Majority Rating and Cumulative T

.313 (p = .005)

Expert Majority Rating and Cumulative P

.339 (p = .002)


5/15/2012

13

Stepwise Regression Analysis

A stepwise regression analysis was conducted to determine h T d P f hi h i ifi di f l


those T and P factors which were significant predictors of low, medium and high risk ratings;

All T and P factors were entered as potential predictors of the dependent risk classification variable, R.


Stepwise Regression Analysis – R


Unstandardised CoefficientsStandardised Coefficients

Predictor B Std. Error Beta

Product subject to a deviation from the relevant international Standard.

9.454 1.247 .438**

Product's standard is recognized as being barely adequate.

6.479 1.899 .189**

Product can be easily converted from a 110 volt product. 4.958 0.821 .361**

Product is likely to be used by unsupervised children. 1.533 0.680 .115*

Product that is used in circumstances where the user is not able to readily disconnect themselves with normal physical reaction to electric shock or burns.

2.830 0.668 .217**

Product is high powered (heat or mechanical energy). 2.782 0.722 .213**

Product that relies on guards and barriers to prevent


8 T and P factors were found to be significant predictors of R, with predictors accounting for over 85% of the variability in scores of R.


Product that relies on guards and barriers to prevent mechanical injury.

3.026 0.958 .161**

Compliance with the applicable Standard is difficult to achieve.

5.78 1.701 .169**

5/15/2012

14

General Conclusions Regarding the Current Risk Engine

> Project Team > Background > Stats Analysis > Risk Index > Expert Analysis > Recommendations

Each of the T and P factors are largely independent and provide an element of unique contribution in identifying the risks associated with electrical products;

A small number of T and P factors have been identified that account for a significant proportion of variability in risk scores and classification;


Results of a cluster analysis has indicated that specific item categories that share similar T and P factor combinations can be identified. These clusters appear to fall in the realms of products classified as low, medium and high risk.


Recommendations for Implementation of the Risk Engine


Recommendation 1 Prior to the Risk Engine being adopted, more detailed

operationalisation is needed for several T and P factors given the ambiguity in their definitions and the risk of subjective interpretation by experts or industry partners who utilise the engine.


5/15/2012

15



Recommendation 2 The Risk Engine should only be employed in conjunction with

expert opinion but should not replace or be a substitute for expert assessment of risk.




Recommendation 3 Given that expert ratings of risk are inconsistent for a significant

proportion of products, further evidence is provided for the combined consultancy of both experts and the Risk Engine in classifying products into risk categories. Such a methodology would enable convergent evidence to be provided for classification of products to categories.


5/15/2012

16



Recommendation 4 Consideration should be given to weighting or removing the

dichotomous nature of certain P and T factors, given their salience in computing an overall risk classification.


Recommendation 5



The diagnostic classification derived from the Risk Engine should be used in conjunction with index scores on metrics that are based on expert assessments of the risk associated with individual products and the need for regulatory control. A methodology is needed to evaluate the consistency between expert ratings of risk and the risk rating provided by the Risk Engine.


5/15/2012

17

Development of


pRisk Engine Mark

2


Aims of Risk Engine Mark 2

Operationalisation of the P and T factor statements to reduce ambiguity;

> Background > Technical Aspects > Application

Improve the dichotomous nature of the P and T factors by introducing a Likert scale;

Develop a framework to evaluate the consistency of an expert panel when rating a new product;

Develop a framework to evaluate the consistency between an


Develop a framework to evaluate the consistency between an expert panel and the Risk Engine.

Develop an web-based application that can be utilised to input ratings from a panel of experts and the Risk Engine for a large volume of products.


5/15/2012

18

Reducing Ambiguity i h P d T


in the P and T Factor Statements


Reducing Ambiguity in the P and T Factor Statements


Recommendation from Risk Engine Mark 1 Evaluation:

“Prior to the Risk Engine being adopted, more detailed operationalisation is needed for several T and P factors given the ambiguity in their definitions and the risk of subjective interpretation by experts or industry partners who utilise the engine ”


who utilise the engine.

5/15/2012

19

Definition of Terms


Generally: Commonly;

Inadequate: Failure or likely failure to ensure electrical safety of user;

Likely: Probably or reasonably foreseeable;

Difficult: Hard to perform or requiring much effort or expense;


Significant: Of importance or of large consequence (in this context this means in terms of the impact of the likely injury or likelihood of misuse causing an injury).




5/15/2012

20




Modifying the P d T F t


and T Factor Statements


5/15/2012

21

Modifying the P and T Factor Statements

P f t t t t th t li i t d


P factor statements that were eliminated: Product is not regulated in Australia;

Product can be easily converted from a 110 volt product.

P factor that was modified:


The dominant supplier’s marketplace applies a standard considered to be inadequate for Australia and New Zealand.


P factor statements that were unchanged:



1. Product generally incorporates new technology2. Product not generally regulated in Asia3. Product subject to a deviation from the relevant international

standard4. The applicable international standard is considered to be

inadequate5 There is a significant compliance cost disincentive


5. There is a significant compliance cost disincentive6. There has been a recent update in the relevant standard7. Compliance with the applicable Standard is difficult to achieve


5/15/2012

22

T factor statements that were unchanged:1 Product that provides an electrical safety function;



1. Product that provides an electrical safety function;2. Product that relies on isolation between LV and exposed ELV parts;3. Product that is likely to be moved during or between uses;4. Product that is used in circumstances where the user is not able to

readily disconnect themselves with normal physical reaction to electric shock or burns;

5. Product that relies on guards and barriers to prevent mechanical injury;


6. Product is likely to be used by unsupervised children;7. Product commonly used in damp locations or where the skin’s

resistance is bypassed;8. Product’s standard is recognized as being barely adequate;9. Product is subject to likely significant misuse;10. Product is high powered (heat or mechanical energy).


T factor statements that were unchanged:11 Product has accessible live parts and relies on safety impedances



11. Product has accessible live parts and relies on safety impedances, current control or cadence for safety;

12. Electrical installation related product, likely to be installed by unskilled persons;

13. Product relies on safety cut-off or interlock for primary safety;14. Product is commonly used locally in an unattended mode but

classified internationally as attended;15. Product that contains high stored energy;


16. Product that has an ionising radiation hazard;17. Product that has hot accessible non-working surfaces;18. Product that has a toxic output;19. Product where a critical failure is not likely to be visible or recognised;20. Product that is generally electrically interconnected with other

products.


5/15/2012

23

What was the Impact of Removing Two P Factor Statements? A sample of 236 products incorporated in the development of

the New Zealand Risk Engine were reviewed;

In total: 134 products met the P factor ‘Product is not regulated

in Australia’ 34 products met the P factor ‘Product can be easily

converted from a 110 volt product’

RMIT University©2011 Risk Engine Validation 45

The overall Risk Engine classification was found to change for 3 products.

On all three occasions, the a High risk classification was changed to a Low risk classification.

An Example of a Product that has Changed Ratings


5/15/2012

24

An Example of a Product that has Changed Ratings


Improving the Dichotomous P and


Dichotomous P and T Factor

Statements


5/15/2012

25

Improving the Dichotomous P & T Factor Statements


Dichotomous P and T factor statements were Initially developed to enable the sum of T and P factor statements to determine the overall level of risk;

Revisions of the Risk Engine have indicated that certain statements are conducive to a dichotomy, whilst others would


be better suited by a Likert scale approach which enables a scaled score based on the level of risk.




Total DichotomousLikert Scale

Possible Combinations

P Factors 8 5 3> 1900

(formerly 189)T Factors 20 12 8


5/15/2012

26

Improving the Dichotomous P & T Factor StatementsP Factors that remained a dichotomy:


P Factors that remained a dichotomy:1. Product generally incorporates new Technology;2. Product not generally regulated in Asia;3. The applicable international Standard is considered to be

inadequate;4. The dominant supplier’s marketplace applies a standard

considered to be inadequate for New Zealand;5. There has been a recent update in the relevant standard.


P Factors that were changed to a Likert scale:1. Product subject to a deviation from the relevant international

standard;2. There is a significant compliance cost disincentive;3. Compliance with the applicable Standard is difficult to achieve.


Improving the Dichotomous P & T Factor StatementsT Factors that remained a dichotomy:1. Product that provides an electrical safety function;


p y ;2. Product that relies on isolation between LV and exposed ELV parts;3. Product that relies on guards and barriers to prevent mechanical injury;4. Product’s standard is recognized as being barely adequate;5. Product is high powered (heat or mechanical energy);6. Product has accessible live parts and relies on safety impedances,

current control or cadence for safety;7. Product relies on safety cut-off or interlock for primary safety;8 P d t i l d l ll i tt d d d b t l ifi d


8. Product is commonly used locally in an unattended mode but classified internationally as attended;

9. Product that contains high stored energy;10. Product that has an ionising radiation hazard;11. Product that has hot accessible non-working surfaces;12. Product that has a toxic output.


5/15/2012

27



T Factors that were changed to a Likert scale:1. Product that is likely to be moved during or between uses;2. Product that is used in circumstances where the user is not able to

readily disconnect themselves with normal physical reaction to electric shock or burns;

3. Product is likely to be used by unsupervised children;4. Product commonly used in damp locations or where the skin’s

resistance is bypassed;5 Product is subject to likely significant misuse;


5. Product is subject to likely significant misuse;6. Electrical installation related product, likely to be installed by unskilled

persons;7. Product where a critical failure is not likely to be visible or recognised;8. Product that is generally electrically interconnected with other products.

Evaluating the Consistency of an


Consistency of an Expert Panel


5/15/2012

28

Evaluating the Consistency of an Expert Panel


Quantitative methodology was utilised to ensure general agreement between experts when evaluating risk level of products;

This methodology is adaptable depending on the number of experts within the panel;


A baseline threshold of 70% agreement was employed. E.g., If a expert panel contained 10 experts, seven

would be required to agree to a certain level of risk prior to a final risk rating being approved.





5/15/2012

29




A Proposed FlowchartSubmission received by ERAC

Submission received by EWG

Information sent to QR-012 for expert assessment

Information inputted into the Risk Engine for assessment

Process completed by EWG

Results of both expert assessment and Risk Engine get presented at QR-012 and discussed


If consensus is not reached, QR-012 discuss the results and another round of voting takes places

This continues until agreement is reached

When agreement is reached, standard is updated

5/15/2012

30

Application for the


Risk Engine Mark 2


Multiple Phases of Data Input

Three phases incorporated within the application to ensure i d d t t b


independent assessment by:

Each individual expert;

The Risk Engine;

The panel of experts as a collective group.


5/15/2012

31

General Homepage



Individual Expert Examination> Background > Technical Aspects > Application


5/15/2012

32





5/15/2012

33

Evaluation of Risk Using the Risk Engine Mark 2






5/15/2012

34




Consistency Evaluation of the Risk Engine and the Panel of Experts



5/15/2012

35







5/15/2012

36

For additional detail regarding the information provided in this

Additional Information


presentation, please contact the project team leaders:

Doctor Adrian Schembri Ph: (03) 9925 6113 Email: [email protected]

Associate Professor Anthony Bedford Ph: (03) 9925 6119


Ph: (03) 9925 6119 Email: [email protected]


CONFIDENTIALITY

IMPORTANT NOTICE

Results and findings contained within this work refer to confidentialinformation. The authors give permission for the distribution of this reportby the Queensland Government only. This report and its findings must betreated in the strictest confidence and not distributed or reproducedwithout the permission of the Queensland Government.


risk engine mark 2: technical aspects and...

Documents