a ssessment instruments for nuclear safety culture : what are we measuring ? antonio c. o. barroso...

ASSESSMENT INSTRUMENTS FOR NUCLEAR SAFETY CULTURE: WHAT ARE WE MEASURING?

Antonio C. O. Barroso IPEN – CNEN/SP

PRESENTATION ROADMAP

Origen of the term IAEA guidance Has the concept permeated? Assessment instruments What are we measuring?

Organizacional Culture

ORIGEN OF THE TERM … CHENORBYL

From the aftermath, at IAEAE

qu

ipm

en

tsS

yst

em

sPro

ced

ure

sC

on

trols

Man

ag

em

en

tPe

op

leA socio-technical system

ORIGEN OF THE TERM - IAEA IN 1986

INSAG1 – The Chernobyl Accident

“A vital conclusion drawn from this behaviour is the importance of placing complete authority and responsibility for the safety of the plant on a senior member of the operations staff of the plant. Of equal importance, formal procedures must be properlyreviewed and approved must be supplemented by the creation and maintenance of a “nuclear safety culture”.

Let’s get the proper understanding of

IAEA GUIDANCE - BASED ON SCHEIN

Monica Haage at ICTP [email protected]

IAEA GUIDANCE - DURING THE 90’S

IAEA INSAG 1 1986

IAEA INSAG 4 1991

IAEA INSAG 7 1992

IAEA SRS 1 1998

IAEA SRS 11 1998

A “definition”“Safety Culture is that assembly of characteristics and attitudes in organizations and individuals which establishes that, as an overriding priority, nuclear plant safety issues receives the attention warranted by their significance”.

IAEA GUIDANCE AND ...MARCH 2011

IAEA TECDOC 1321 2002

INSAG Series 15 2002

IAEA SRS 74 2012

IAEA TECDOC 17072013

2011-03-11

HAS THE CONCEPT PERMEATED?

Humans alone and in groups are the root agents of organizations and, as such, we could build upon the definition stated in INSAG 4

Safety Culture is the assembly of systems, characteristics, mindset and attitudes at the organizational and individual levels which assures that: (a) as an overriding priority, nuclear safety issues

receive the attention warranted by their significance; and

(b) adequate resources, information and actionable knowledge are empowered at the decision / action points where safety issues are dealt with.

HAS THE CONCEPT PERMEATED? A BIBLIOMETRIC VIEW

From Scopus data base until Dec/2011From 1997 -2011: nuclear safety culture

seemed to be left out of the agenda of the nuclear industry.

In relevant journals and conferences, the interest on the subject has been near stagnant, both in terms of publications and citations.

At the same time for the other areas taken together, there is a vigorous increasing interest on the subject


Safety culture in the nuclear area

years Publications Citations

Average Slope Average Slope

First period 1990-2000 1.82 -0.08 4.91 -0.91

Second period 2001-2011 7.45 0.06 0.55 -0.05

Overall 1990-2011 4.64 0.38 2.73 -0.42

Safety culture outside the nuclear area



First period 1988-94 4.14 0.86 0.00 0.00

Second period 1995-2003 27.67 4.58 55.22 19.13

Third period 2004-11 170.88 20.30 1078.63 282.44

Overall 1988-2011 68.54 10.19 380.25 71.13


It could be unfair to compare numbers of nuclear area to all others, but it is wise and fair to compare the growth rates

Safety culture in the nuclear area



Overall 1990-2011

4.64 0.38 2.73 -0.42

Safety culture outside the nuclear area



Overall 1988-2011 68.54 10.19 380.25 71.13

Are we happy with that?

ASSESSMENT INSTRUMENTS

I don’t agree that “What You Can't Measure, You Can't Manage”, but I

do agree that “What You Can Assess, You Can Better Manage”

IAEA documents provide thorough, diverse and valuable guidance, but they are only sketchy concerning quantitative assessment models. Note that there are other assessment methods and we need them all, but I am going to focus on the quantitative ones.

ASSESSMENT INSTRUMENTS - REFLECTIVE OR FORMATIVE

Is safety culture a reflective or formative construct?

Health Diet (reflective) Health conservation (formative)

Health Diet

Health Cons.

ASSESSMENT INSTRUMENTS - REFLECTIVE OR FORMATIVE

Blood alcohol content

# Wine glasses

# beer cans

# vodka drinks

# whisky doses

Drunkness level

Blow test Physical reaction test

Clocked memory test

Observable variables (indicators) not necessarily correlated

(should be complete)

Observable variables (indicators) that should be correlated

(some are enough)

Formative construct Reflective construct

ASSESSMENT INSTRUMENTS – RELIABILITY AND VALIDITY

ASSESSMENT INSTRUMENTS – RELIABILITY AND VALIDITY

Dimensionality of the construct Content validity Convergent validity Discriminant validity Predictive validity Reliability

“validation is cumulative”multiple studies, different settings …

ASSESSMENT INSTRUMENTS - “VALIDATED” MODELS OF NUCLEAR SAFETY CULTURE?

Web of science – Apr/2014

WHAT ARE WE MEASURING?

Article 1 - LEE, T. Assessment of safety culture at a nuclear reprocessing plant. Work and Stress, v. 12, n. 3, p. 217-237, 1998.Safety culture assessment of Sellafield reprocessing plant focusing safety related attitudes. Initial questionnaire had 172 itens and gather data from 5926 participants. Initially EFA by PCA resulted in 38 factors, but later reduced to 19 and 81 items. Predictive validity was done against the number of reported accidents with significant time loss.The model was not further evolved and it would be unpractical by todays standards


Article 2 - Lopes de Castro, B.; Gracia, F. J.; Peiró, J. M.; Pietrantoni, L.; Hernández, A. Testing the validity of the International Atomic Energy Agency (IAEA) safety culture model. Accident Analysis and Prevention, v. 60, p. 231-244, 2013. They have used as model the 5 characteristics (factors) and 37 attributes (indicators) recommended by AEA Safety Report Series no. 42 (2005) and have tried to validate it.The model failed to present adequate psychometric capacity. Most of the indicators were not good manifestations of their factors, low face validity and only moderate content validity. Most alarming was the lack of discriminating validity.


• Not much, or • Not nuclear safety culture

• But other segments are!• For instance in Hospitals and similar

organizations many steps of the “cumulative” validation have been done• a model for patient safety culture, developed

in 2005, has been applied, adapted and validated in different settings (10) and countries (5)

a ssessment instruments for nuclear safety culture : what are we measuring ? antonio c. o. barroso...

Documents