evaluation framework - safework.nsw.gov.au · evaluation plan the mechanics of how a particular...
TRANSCRIPT
EVALUATION FRAMEWORKWORK HEALTH AND SAFETY ROADMAP FOR NSW 2022
JULY 2017
DisclaimerThis publication may contain information about the regulation and enforcement of work health and safety in NSW. It may include some of your obligations under some of the legislation that SafeWork NSW administers. To ensure you comply with your legal obligations you must refer to the appropriate legislation.Information on the latest laws can be checked by visiting the NSW legislation website www.legislation.nsw.gov.auThis publication does not represent a comprehensive statement of the law as it applies to particular problems or to individuals or as a substitute for legal advice. You should seek independent legal advice if you need assistance on the application of the law to your situation.This material may be displayed, printed and reproduced without amendment for personal, in-house or non-commercial use.
Catalogue No. SW08641SafeWork NSW, 92–100 Donnison Street, Gosford, NSW 2250Locked Bag 2906, Lisarow, NSW 2252 | Customer Experience 13 10 50Website www.safework.nsw.gov.au© Copyright SafeWork NSW 0817
ContentsABBREVIATIONS AND ACRONYMS 6
KEY EVALUATION TERMS AND DEFINITIONS 7
EXECUTIVE SUMMARY 9
1. INTRODUCTION 10
1.1. Overall objectives of evaluating the Roadmap 10
1.2. Purpose of the framework 10
1.3. Overview of the Roadmap 10
1.4. Alignment with the NSW Government Program Evaluation Guidelines (2016) 11
2. EVALUATION GUIDELINES 12
2.1. Principles of evaluation 12
2.2. Information management 13
2.3. Ethical conduct of evaluation activities 14
2.4. Risk management 14
2.5. Updating the evaluation framework 14
3. FRAMING THE ROADMAP EVALUATION ACTIVITIES 15
3.1. Theory of change – how the Roadmap is expected to work 15
3.2. Logic models and outcomes matrices frame the evaluation 16
3.2.1. Outcomes matrices and performance measures for Action Area 1: Embed the health and safety landscape in NSW workplaces 20
3.2.2. Outcomes matrices and performance measures for Action Area 2: Prioritise sectors, harms, workers and workplaces where the most significant WHS risks exist 25
3.2.3. Outcomes matrices and performance measures for Action Area 3: Build exemplar regulatory services 29
3.2.4. Outcomes matrix for long term outcomes 33
3.3. Roadmap evaluation and guiding questions 34
4. PERFORMANCE MONITORING 39
4.1. About performance monitoring 39
4.2. Collection of performance metrics 40
4.2.1. Reporting performance metrics 40
4.3. Accountability 40
5. EVALUATION GOVERNANCE, EVALUATION PLANNING AND ADVICE ON METHODS 41
5.1. Governance and management of evaluation studies 41
5.2. Directorate-level annual evaluation planning 42
5.3. Individual project evaluation plans 42
5.4. Decision support tool 42
5.5. Evaluation checklist 45
5.6. Methodological approaches 46
6. UNDERSTANDING, USING AND REPORTING EVALUATION FINDINGS 48
6.1. Using evaluation results for service and product improvement 48
6.2. Communication through evaluation reports 48
6.3. Analysis of attribution and causal contribution 49
6.4. Interpreting trends in lag indicators (long term policy outcomes) 50
Tables and figuresTABLES
Table 1: Action Area 1 Inputs Outcomes Matrix 20
Table 2: Activities, reach, short term outcomes and medium term outcomes 20
Table 3: Action Area 2 Inputs Outcomes Matrix 25
Table 4: Activities, reach, short term outcomes and medium term outcomes 25
Table 5: Action Area 3 Inputs Outcomes Matrix 29
Table 6: Activities, reach, short term outcomes and medium term outcomes 29
Table 7: Outcomes matrix for Roadmap long term outcomes 33
Table 8: The Roadmap Evaluation Priority Levels 43
Table 9: Commonly used methods in evaluation 47
FIGURES
Figure 1: Overarching WHS Roadmap for NSW program logic model 18
Figure 2: Logic model for Action Area 1: Embed the health and safety landscape in NSW workplaces 19
Figure 3: Logic model for Action Area 2: Prioritise sectors, harms, workers and workplaces where the most significant WHS risks exist 24
Figure 4: Logic model for Action Area 3: Build exemplar regulatory services 28
Figure 5: Evaluation checklist 45
6 SAFEWORK NSW
Abbreviations and acronyms
ACT Australian Capital Territory
AWU Australian Workers’ Union
CFMEU Construction, Forestry, Mining and Energy Union
NSW New South Wales
PCBU Persons conducting a business or undertaking. The Work Health and Safety Act 2011 places the primary duty of care and various other duties and obligations on a 'person conducting a business or undertaking'; the meanings are set out in Section 5 of the Act.
The Roadmap Work Health and Safety Roadmap for NSW 2022
SafeWork SafeWork New South Wales
SW NSW SafeWork New South Wales (used as abbreviation in tables)
WHS Work health and safety
WHS ROADMAP EVALUATION FRAMEWORK 7
Key evaluation terms and definitions
Evaluation A rigorous, systematic and objective process to make an evidence-based assessment of the effectiveness, efficiency, appropriateness and sustainability of an initiative.
Evaluation design How a particular evaluation is set up to provide the evidence to achieve its purpose (eg experimental design). The design takes account of the appropriateness of methods for the project’s scope and status, as well as ethical safeguards.
Evaluation framework A structure to guide evaluation and monitoring activities, based on systematic and conceptual thinking about what a program is setting out to achieve and how the anticipated outcomes can be assessed.
Evaluation plan The mechanics of how a particular evaluation will be carried out. It sets out the tasks and responsibilities to implement an evaluation project/study and the budget.
Evaluation strategy An overall approach to evaluation that reflects the strategic priorities and state of the project at the time. It starts with the purpose (why), focus (what), and stakeholder information needs (what evidence, by when). It references the project outcomes framework and links with other policy frameworks/strategies. It includes the main evaluation activities/studies, with an outline of designs and methods. It also sets out the timing, responsibilities and resources.
Inputs The resources (human and other) you use to do the work.
Outcome A result that can be measured or observed and a changed condition of people/organisations/systems.
Reach Used in program logic to show which group/s are being targeted by particular activities
Short term outcomes Changes expected to occur as a direct result of your work.
Theory of change These are the big research-based theories about how change occurs for individuals, groups, organisations and communities. Theories of change can provide a conceptual framework for developing or critiquing a program theory.
Medium term outcomes Changes your program is expected to contribute to, the next link in the causal chain of outcomes after short term outcomes. Generally, they relate to the strategic objectives of a program.
Long term outcomes Changes your program is expected to contribute to, the next link in the causal chain of outcomes after medium term outcomes.
Outputs The tangible products or services the activity will deliver.
Outcomes matrix A matrix of project outcomes at different levels, causally linked by a results logic diagram. For each outcome, it shows attributes of success and is usually used to frame an evaluation strategy and/or a monitoring system.
8 SAFEWORK NSW
Performance monitoring Provides information about whether or not an initiative is on track. That is, information about how much has been done to what quality and sometimes about the immediate outcomes achieved. Performance monitoring is one source of evidence for an evaluation study but monitoring by itself is not evaluation and is less in-depth than evaluation. Monitoring is ongoing while evaluation is typically periodic; mainly uses data captured as part of delivering services; informs quality assurance activities and is used to manage programs/businesses on an ongoing basis.
Program theory An explicit statement of how an intervention/program is understood to contribute to observed or intended outcomes.
Program logic model A diagram that shows how an intervention/program is assumed to work by causally linking program activities with intended outcomes.
Purposive sampling A sample of subjects selected deliberately (on purpose) because these subjects have particular characteristics of interest.
WHS ROADMAP EVALUATION FRAMEWORK 9
Executive summaryThis is the evaluation framework for the Work Health and Safety Roadmap for NSW 2022 (the Roadmap). The evaluation framework is intended to provide a blueprint for ongoing evaluation of the three action areas over the life of the Roadmap and it will be reviewed and updated in line with the two-yearly review of the Roadmap itself.
The development of the evaluation framework has been a collaborative process. Six key informants representing unions and business peak groups provided input into the framework through interviews. SafeWork Directors and members of the Business Performance team contributed via interviews and workshops. This consultative process, complemented by a literature review, Research into features of effective evaluation frameworks, was delivered in February 2017.
The framework has an overarching program logic and program logics for each of the three action areas under the Roadmap. The action area program logics have been used to specify outcomes matrices that represent the intended outcomes of the action areas at different levels, the attributes of success and potential data sources.
The framework identifies key evaluation and guiding questions (for each action area) and includes a decision support tool to identify priorities for evaluation. The framework also covers evaluation governance, advice on planning evaluations, and advice about reporting and using findings to inform and influence service improvement. In addition, evaluation guidelines outline principles for evaluation and identify ethical considerations.
10 SAFEWORK NSW
1. Introduction1.1. Overall objectives of evaluating the RoadmapEvaluation can be defined as a rigorous, systematic and objective process to assess a program’s effectiveness, efficiency, appropriateness and sustainability. An effective evaluation produces information (within ethical standards) that key stakeholders find credible, relevant and useful. Evaluation studies need to focus on the aspects of the program that stakeholders find most relevant at a particular time, and to function within the limits of resources and timelines.
The objectives of evaluating the Roadmap are to:
1. Measure whether there is a change in line with the outcomes identified in the program logics for each action area.
2. Assess whether the development, design and implementation of activities under the Roadmap are effective, and which activities need improvement.
3. Assess the extent to which activities under the Roadmap contribute to the achievement of outcomes shared with industry, government and community partners.
1.2. Purpose of the frameworkThis customised evaluation framework is intended to guide evaluation and monitoring for the Work Health and Safety Roadmap for NSW 2022 (the Roadmap) over the next six years. The evaluation framework will provide SafeWork NSW guidance about:
• when and how the Roadmap will be evaluated
• the development of sound evaluation plans and the implementation of monitoring and evaluation activities that ensure SafeWork can determine whether key objectives of the Roadmap have been met and to what extent
• the development of a risk-based operating model where evaluation effort is prioritised to the most critical area.
Formal project evaluation is used to provide a systematic assessment of a program’s efficiency and effectiveness in delivering intended outcomes. The key point is that evaluation is also about learning and program improvement. A high quality evaluation can provide information about the success of a program of work, the factors that have contributed to success and failure, and the challenges that have had to be overcome along the way. A solid evaluation informs ongoing improvement.
The evaluation framework was developed in consultation with SafeWork Directors and representatives of peak industry bodies and associations.
1.3. Overview of the RoadmapThe Roadmap is a project under the NSW Government’s Better Regulation Division (BRD) Strategic Plan 2016, and is the responsibility of SafeWork NSW. It is a six-year strategy that will drive state-wide activities for improvement in work health and safety; the BRD measure of success is to reduce the incidence rate of fatalities and serious injuries in NSW workplaces.
The Roadmap is an agreement aimed not only at the regulator, but at peak industry and employee bodies/associations, community leaders, and each employer and worker, all of whom have played a major part in contributing to the Roadmap through extensive consultation.
WHS ROADMAP EVALUATION FRAMEWORK 11
The Roadmap’s vision is for healthy, safe and productive working lives, and it sets out targets to measure success. Its purpose is to ‘drive state-wide activities for improvement in work health and safety in NSW’. The Roadmap identifies three action areas:
1. Embed the health and safety landscape in NSW workplaces
2. Prioritise sectors, harms, workers and workplaces where the most significant WHS risks exist
3. Build exemplar regulatory services.
An underlying enabler is community commitment, which is seen as important to the achievement of the three action areas. Community means everyone – workers, employers, professional and community groups, and regulators. A key factor for the Roadmap is that SafeWork is a contributor to many workplace health and safety actions which they do not lead.
The Roadmap spans 2016–2022 but it will be refreshed and released every two years to ensure it stays relevant. The Roadmap combines existing workplace health and safety work with new actions. The first year of the Roadmap has been dedicated to planning the delivery of, and operationalising, the action areas.
1.4. Alignment with the NSW Government Program Evaluation Guidelines (2016)Across all agencies, the NSW Government is committed to using evaluation to inform policy, and achieve greater efficiency and transparency in government, through the use of digital services and data analytics.
The NSW Government Program Evaluation Guidelines (2016) outline the government’s approach to program and agency evaluations.
12 SAFEWORK NSW
2. Evaluation guidelinesThis section provides information about how best to conduct, commission and manage evaluation activities.
Using independent evaluators and research groups to carry out the different evaluation phases may be advantageous because their independence has the potential to enhance the credibility of the findings to program stakeholders at a strategic policy level. Evaluation activities might be able to be conducted internally depending on available internal resources and expertise.
2.1. Principles of evaluation1
The following evaluation principles have informed the design of the evaluation framework and aim to ensure that all evaluation activities of the Roadmap are of a high quality and will provide useful evidence. The principles are drawn from the NSW Government Program Evaluation Guidelines (2016), with additional SafeWork NSW principles to guide individual evaluation planning.
NSW Government Program Evaluation Guidelines Principles1. Build evaluation into your program design. Plan your evaluation before you implement a project
to ensure the project has clearly defined and measurable outcomes and sources of information have been identified and data collection systems are in place. This in turn increases the quality of the evaluation.
2. Base your evaluation on sound methodology. Use best practice methodologies to suit the program’s size, significance and risk.
3. Include resources and time to evaluate. Consider the required evaluation resources and timeframe when planning the project. Ensure evaluation findings will be available when needed to support decision-making.
4. Use the right mix of expertise and independence. Choose evaluators who are experienced and independent from program managers, but always include program managers in evaluation planning. An external evaluation can bring expertise, capacity and credibility based on greater independence. An internal evaluation can bring first-hand knowledge about a policy or program, and develop capacity for future evaluation work. A hybrid model – such as an evaluation conducted internally with an external evaluation partner or mentor – can be a cost-effective way of strengthening an internal evaluation.
5. Ensure proper governance and oversight. Use governance processes to ensure oversight of evaluation design, implementation and reporting.
6. Be ethical in design and conduct. Carefully consider the ethical implications of any evaluation activity, particularly collecting and using personal data, and any potential impacts on vulnerable groups. You may need formal review and approval from an ethics committee certified by the National Health and Medical Research Council.
7. Be informed and guided by relevant stakeholders. Listen to stakeholders, including program participants, government or non-government staff involved in managing and delivering the program, and senior decision makers.
8. Consider and use evaluation data meaningfully. Include clear statements of findings for consideration in evaluation reports. Use reports to inform any decisions about changes to programs.
9. Be transparent and open to scrutiny. Publicly release key information about all aspects of the evaluation unless there is an overriding public interest against disclosure. This could include methodologies, assumptions, analyses and findings.
1 NSW Government Program Evaluation Guidelines (2016).
WHS ROADMAP EVALUATION FRAMEWORK 13
10. Be clear about what success could look like. Successful evaluation is reliant on clear problem definition. You may either develop a program logic for your specific intervention, referencing the action area program logics, and/or identify attributes of success for your specific activity that fit under one or more specific outcomes in the three action area logics.
11. Balance the technical, strategic and political dimensions in planning evaluations. Evaluation is often seen as a technical exercise. But evaluation is also a strategic exercise that aims to deliver information to key stakeholders when they need to make decisions. For SafeWork, the (small p) political dimension is working with your industry partners who may have different interests in the program and the evaluation, including different ideas about evidence, methods, and how the evaluation is managed and reported.
12. Make use of lead indicator and other administrative data already collected through administrative data systems.
2.2. Information managementAll data, information and documentation provided or obtained during the course of any evaluation activities should be kept in strict confidence and in accordance with the Australian Privacy Principles effective 12 March 2014, which cover the collection and management of personal information.
• Information should be sought from informants lawfully, with their consent as appropriate, and used only for the purposes of the evaluation.
• Informants should be advised of the intended primary and secondary use for the collected information.
• Information and documents must be kept in secure places and not divulged to any party other than those involved in contracting or conducting the evaluation. All evaluation data should be kept on a secure, password-protected server and be protected by protocols to ensure the security of any data transmitted through the server to other locations. For example, auditable, granular data access for evaluation folders (limited to relevant persons), firewall protection and intrusion detection system, nightly backup to an offsite location and strong physical security (double locks and limited physical access to servers). All mobile devices used to access and transmit project data should be controlled centrally so these can be wiped remotely if necessary.
• Information must be kept only for the purposes of the evaluation, record-keeping, or research validation, and should be de-identified within the terms of the Australian Privacy Principles’ application to evaluation. Evaluation information/data should be kept for a minimum of ten years before being destroyed.
External contractors for evaluation activities should be asked to describe the following.
• How will data be stored?
• The format of stored data.
• Arrangements for security of stored data, including when and by whom that shared data can be accessed.
• The duration the data will be stored.
• The method of destruction of data.
14 SAFEWORK NSW
2.3. Ethical conduct of evaluation activitiesIt is important that SafeWork staff members including those commissioning evaluation studies and any evaluation contractors consider the ethics of collecting and storing evaluation data and be guided about ways to solve ethical dilemmas that may arise during the evaluation. In your evaluation planning you should consider how consent will be obtained.
The implementation of and commissioning of, and all SafeWork evaluation activities should be guided by the Australasian Evaluation Society (AES) Guidelines for the Ethical Conduct of Evaluations, the NHMRC National Statement on Ethical Conduct in Human Research and the NHMRC Ethical Considerations in Quality Assurance and Evaluation Activities (2014).
Triggers for ethical review
The NHMRC Ethical Considerations in Quality Assurance and Evaluation Activities (2014) provides advice about triggers for ethical review, and relevant triggers are:
• where the activity potentially infringes the privacy or professional reputation of participants, providers or organisations
• secondary use of data – using data or analysis from quality assurance or evaluation activities for another purpose
• comparison of cohorts
• targeted analysis of data involving minority or vulnerable groups, whose data is to be separated out of that data collected or analysed, as part of the main quality assurance or evaluation activity.
2.4. Risk managementEvaluation studies of Roadmap initiatives will need to deal with a number of issues and risks for implementation. These fall into two main areas.
1. Availability, reliability and usefulness of expected administrative data.
2. Gaining the desired level of participation in data collection activities across different sectors, types of businesses and stakeholder groups.
The extent that these issues will influence the capacity of the evaluation studies to adequately answer the evaluation questions depends on their resolution and on strategies to engage and communicate with stakeholders.
The evaluation plan for each phase should include a risk management strategy, categorisation of risks using a risk matrix, and risks and solutions should be reported against regularly in progress reports.
2.5. Updating the evaluation frameworkThe evaluation framework is based on consultations about how the Roadmap is expected to work (theory of change), be used, and produce outcomes. The framework may need to be refined when:
• two-year evaluation cycles show that the program works in ways not explained by the current theories of change
• the program processes and implementation processes are changed to increase the effectiveness of activities under the Roadmap
• the evaluation framework is not producing useful information.
WHS ROADMAP EVALUATION FRAMEWORK 15
3. Framing the Roadmap evaluation activities
This chapter describes an overarching program logic model for the Roadmap and logic models for each of the action areas. The action area logic models describe how these can be used to frame evaluations of the Roadmap over the six years, drawing upon the underlying theories of change for the work. The logic models were used to develop the key evaluation questions about the Roadmap.
To guide evaluation, outcomes matrices have been developed for each action area, which set out the measures and potential data sources for each level of the logic models.
3.1. Theory of change – how the Roadmap is expected to workTheories of change are useful to explain how and why a change is expected to happen in a particular context. Because the Roadmap is a multi-faceted intervention combining educative, environmental and regulatory approaches that have strategic, tactical and operational foci, it is useful to identify one or more broad theories of change that are built upon how similar interventions have worked in other circumstances. The outcomes logics reference some elements of each of these theories of change.2 In practice most of our programs have one or more elements of each of these theories. A typical approach would be to gain an understanding of the behaviours of a target group and then seek to change these behaviours using visits, doing compliance checks and providing guidance materials (theory of behaviour and theory of reasoned action). This approach is often supported by social media campaigns (diffusion theory). In short, most of our programs are based on the idea that if we make it easy for people to comply then they are more likely to do so.
1. Theory of behaviour influenced by social, cognitive and emotional influences. The NSW Department of Premier and Cabinet has invested in the development of a newer understanding of the way people behave, in an attempt to gain better compliance with a wide range of behaviours. Their work has ranged from changes in no-show rates at public hospitals and increasing quitting rates amongst smokers to increased compliance with motor vehicle laws. The principle behind the Behavioural Insights Unit’s work is that behavioural change is not exclusively influenced by rational choices; the Behavioural Insights Unit draws on research into behavioural economics and psychology to influence choices in decision-making. By focusing on the social, cognitive and emotional behaviour of individuals and institutions it suggests that subtle changes to the way decisions are framed and conveyed can have big impacts on behaviour.3
2. Diffusion theory holds that change occurs when an innovation (new practice/idea/approach) is diffused and adopted leading to certain consequences. Diffusion is a process in which an innovation is communicated and exchanged through social channels such as media campaigns and networks. Diffusion theory has four key concepts and related theories:
• Theory of perceived attributes, which states than an individual will adopt an innovation if they perceive it has a relative advantage over an existing approach; it is compatible with existing values and practices; it is not too complex; it can be tested for a limited time without committing to adoption and with observable results.
• Communication channels, that is, propositions about how messages are shared effectively. Mass media is useful for contacting large numbers of people and increasing awareness. Interpersonal contact is more effective for persuading people to accept a new idea. Effective communication can depend on engagement of peers and opinion leaders.
• Diffusion among members of a social system. Diffusion is affected by social structure, system norms, opinion leaders and change agents.
• A time dimension to diffusion. Innovation decision process theory postulates that the adoption process has five steps: 1) Knowledge 2) Persuasion 3) Decision 4) Implementation, and 5) Confirmation. Further, those individuals differ in when they will adopt an innovation in predictable ways.
2 Funnell SC and Rogers PJ (2011). Purposeful Program Theory. Jossey-Bass. 1st Edition.
3 http://bi.dpc.nsw.gov.au/
16 SAFEWORK NSW
3. Theory of reasoned action and theory of planned behaviour. These theories are based on the notion that humans are rational and have control over what they do. Intentions predict behaviours, and intentions are influenced by a person’s beliefs about the likely consequences of the behaviour, their attitudes towards the behaviour and consequences and perceptions of norms, that is, other people’s opinions of the behaviour.
3.2. Logic models and outcomes matrices frame the evaluationThe Roadmap is intended to drive state-wide activities for improvement in work health and safety – so that NSW residents have healthy, safe and productive working lives. The Roadmap comprises three action areas to help achieve SafeWork’s targets.
The overarching program logic (Figure 1) shows how the three action areas are expected to contribute to medium and long term outcomes and provides a line of sight for the Roadmap implementation. Three pipeline logic models, one for each action area, are also shown with more detail about the activities and the causal links between activities and specific outcomes (Figures 2, 3 and 4). The action area logic models provide the structure to prescribe the scope of evaluation activities and focus the data collection methods.
Under Action Area 1, the problem being addressed by SafeWork is that businesses need assistance to embed the health and safety landscape. Under Action Area 2, the problem being addressed is that SafeWork needs to work with industry to identify and address high risk sectors and harms, protect at risk workers and ensure that high risk workplaces meet compliance standards. Under Action Area 3, the problem being addressed is that SafeWork needs to provide right touch regulatory approach so that customers can be confident that SafeWork will enforce WHS to protect workers, that SafeWork will use data and insights to make decisions and provide innovative services to meet the needs of business.
What is a program logic model?
A program logic model is an analysis of the aims, objectives and activities of an initiative (in this case the Roadmap) that represents ideal ‘outcomes’ at different levels and stages, and the causal links between them. It sets out the chain of expected outcomes in a diagram, starting with the inputs over which you have a high degree of control and progressing to the hoped for long term outcomes. It shows short term outcomes and medium term outcomes that logically, if these are achieved, should then lead to the longer term outcomes. A program logic diagram is not a process flow diagram or a systems map even though it can resemble one.
How to read the program logic model4
Certain resources are needed to deliver your projects and services
If you have access to them, then you can use them to accomplish your planned activities
If you accomplish your planned activities, then you will hopefully deliver the services/products (amount, quality) that you intended and reach your intended audience
If you accomplish your planned activities to the extent you intended, then your audience will benefit in certain ways
If these benefits to participants are achieved, then certain changes in workplaces/systems and impacts on individuals and the community might be expected to occur
Inputs
Your planned work Your intended results
Activities Reach Outcomes Long termoutcomes
1 2 3 4 5
4 W.K. Kellog Foundation Logic Model Development Guide
WHS ROADMAP EVALUATION FRAMEWORK 17
Assumptions
The Roadmap overarching program logic is underpinned by a number of assumptions about how the Roadmap will work.
• The communications campaign will result in increased engagement with SafeWork NSW.
• Interacting with SafeWork NSW will help businesses become more confident about, and capable of, managing WHS in NSW.
• SafeWork can work collaboratively with stakeholders in order to develop and deliver the right products and services.
• SafeWork NSW will have sufficient and consistent resources to deliver on the Roadmap.
• Engagement with industry partners (peak bodies, associations, employers and employees) will result in them taking actions in line with the Roadmap, and their actions will contribute to achieving the long term safety outcomes.
• SafeWork NSW will put systems and processes in place to deliver exemplar regulatory services.
• A risk-based regulatory approach will improve compliance and workplace practices in targeted industries and workplaces. Inspections improve compliance.
• A right touch regulatory approach leads to awareness and action.
The achievement of outcomes under the Roadmap will be influenced by factors external to SafeWork that are outside of the control of the agency. According to SafeWork stakeholders, external factors include the following.
• Change in government/change in priorities for the government
• Changes in legislation/regulatory framework
• Economic environment and activity. Industry/business activity trends with industries expanding or contracting. For example, work health and safety can be compromised when businesses are rapidly expanding.
• Changes in technology and other business operating conditions
• Changes in workforce structure and employment conditions. Two trends were raised: 1) the increase in casual or temporary employment 2) more people working as contractors instead of employees, for example in the road transport context.
• Changes in the nature of work (such as more automation of dangerous work), the mix of work (such as a shift from manufacturing to less dangerous service provision as the mainstay of the economy).
Part of the function of evaluation studies will be to test the assumptions and also take account of the influence of external factors when analysing and interpreting results.
Outcomes matrices show what information can be used to assess the achievement of outcomes
Outcomes matrices set out the full range of information needed to assess the achievement of outcomes and so answer the evaluation and guiding questions, and indicate where the information can be sourced from.
Outcomes matrices include:
• attributes of success – what outcomes should look like when achieved
• performance evaluation measures – metrics for attributes of success.
18 SAFEWORK NSW
Fig
ure
1: O
vera
rch
ing
WH
S R
oad
map
fo
r N
SW
pro
gra
m lo
gic
mo
del
Bus
ines
s p
rob
lem
Act
ivit
ies
Rea
chSh
ort
-ter
m o
utco
mes
Med
ium
-ter
m o
utco
mes
Long
-ter
m o
utco
mes
Targ
eted
sta
keho
lder
sar
e aw
are
of
WH
S is
sues
ac
coun
tab
iliti
esIn
crea
sed
will
ing
ness
to
take
ac
tion
Targ
eted
sta
keho
lder
s id
enti
fy
way
s to
imp
rove
WH
S in
the
ir
ind
ustr
ies,
wo
rkp
lace
sW
ork
pla
ces
und
erst
and
the
d
iffer
ent
elem
ents
of
H&
S
land
scap
e, r
esp
onsi
bili
ties
Targ
eted
bus
ines
ses
are
awar
e of
R
TW is
sues
/acc
oun
tab
iliti
es a
nd
iden
tify
way
s to
imp
rove
RT
W
out
com
es in
the
ir w
ork
pla
ceIn
crea
sed
aw
aren
ess
of
the
Safe
Wor
k b
rand
Incr
ease
d e
ngag
emen
t w
ith
Saf
eWo
rk
Bus
ines
ses
need
ass
ista
nce
to e
mb
ed t
he h
ealth
and
sa
fety
land
scap
e
Eff
ecti
ve d
eliv
ery
of
Act
ion
Are
a 1.
Em
bed
the
hea
lth
and
sa
fety
land
scap
e in
NSW
wor
kpla
ces
Pea
k in
dus
try
bo
die
s,
asso
ciat
ions
, co
mm
unit
y le
ader
s, N
SW g
ener
al
com
mun
ity,
em
plo
yers
and
w
ork
ers
NSW
wo
rkp
lace
s ha
ve
hig
h sa
fety
sta
ndar
ds
A w
ell i
nfo
rmed
NSW
co
mm
unit
y,
know
led
gea
ble
ab
out
W
HS
issu
esA
ll jo
bs
are
des
igne
d
wit
h sa
fe w
ork
p
ract
ices
and
co
ntro
lsE
ffec
tive
RT
W
pra
ctic
es a
re
emb
edd
ed in
NSW
w
ork
pla
ces
Few
er fa
talit
ies,
se
rio
us in
juri
es,
illne
ss a
nd
mus
culo
skel
etal
d
iso
rder
sat
wo
rk
Gen
eral
co
mm
unit
y, e
mp
loye
rs a
nd
wo
rker
s ac
cess
pro
duc
ts a
nd
serv
ices
Incr
ease
d c
onfi
den
ce, c
apab
ility
and
w
illin
gne
ss t
o t
ake
acti
on
Em
plo
yers
pro
acti
vely
imp
lem
ent
elem
ents
of
H&
S la
ndsc
ape
and
im
pro
ve t
heir
sys
tem
s an
d c
ont
rols
Wo
rker
s im
pro
ve t
heir
wo
rk p
ract
ices
Targ
eted
sta
keho
lder
s ad
voca
te fo
r im
pro
ved
WH
S w
ithi
n th
eir
own
sect
ors
Ind
ustr
ies
pro
acti
vely
imp
lem
ent
sect
or
wid
e W
HS
init
iati
ves,
red
uce
wor
kers
’ exp
osur
e to
ris
kTa
rget
ed e
mp
loye
rs a
nd w
ork
ers
acce
ss s
ervi
ces
Targ
eted
bus
ines
ses
imp
lem
ent
RT
W
pra
ctic
es in
the
ir w
ork
pla
ces
Imp
rove
d a
war
enes
s o
f hi
gh
imp
act
harm
s an
d W
HS
issu
es o
f ta
rget
ed in
dus
trie
s an
d w
ork
ers
Incr
ease
d c
onfi
den
ce, c
apab
ility
an
d w
illin
gne
ss t
o t
akes
act
ion
PC
BU
s aw
are
of
Saf
eWo
rk’s
re
gul
ato
ry, a
dvi
sory
and
en
forc
emen
t ap
pro
ach;
pro
duc
ts
and
ser
vice
s; la
test
dev
elo
pm
ent/
s ad
vanc
es in
WH
SP
CB
Us
und
erst
and
the
ir W
HS
re
spo
nsib
ilite
s an
d w
hat
com
plia
nce
loo
ks li
keP
CB
Us
and
key
sta
keho
lder
s re
cog
nise
the
ben
efits
of
wo
rkin
g
wit
h S
afeW
ork
to
dev
elo
p
inno
vati
ve s
olu
tio
nsS
afeW
ork
use
dat
a sy
stem
s an
d
evid
ence
to
info
rm
dec
isio
n-m
akin
gS
afeW
ork
pub
lish
info
rmat
ion/
stan
dar
ds/
po
licie
s In
tera
ctio
ns w
ith
cust
om
ers
mee
t ex
pec
ted
sta
ndar
ds
of
serv
ice
Saf
eWo
rk n
eed
s to
wo
rk
wit
h in
dus
try
to id
enti
fy
and
ad
dre
ss h
igh
risk
se
cto
rs a
nd h
arm
s,
pro
tect
at
risk
wo
rkp
lace
s m
eet
com
plia
nce
stan
dar
ds
Eff
ecti
ve d
eliv
ery
of
Act
ion
Are
a 2.
Pri
orit
ise
sect
ors,
har
ms,
w
orke
rs a
nd w
orkp
lace
s w
here
mos
t si
gni
fica
nt
WH
S ri
sks
exis
t
Pea
k in
dus
try
bo
die
s,
asso
ciat
ions
, co
mm
unit
y le
ader
s, m
anag
emen
t an
d
staf
f o
f ta
rget
ed N
SW
Gov
ernm
ent
agen
cies
, m
anag
emen
t an
d s
taff
of
sele
cted
hig
h ri
sk
wo
rkp
lace
s
Hig
h ri
sk in
dus
trie
s ar
e si
gni
fican
tly
safe
r an
d h
ealt
hier
Hig
h im
pac
t ha
rms
are
elim
inat
ed o
r si
gni
fican
tly
red
uced
A s
afe
and
hea
lthy
N
SW p
ublic
sec
tor
Saf
e an
d h
ealt
hy
wo
rker
sS
afe
and
hea
lthy
w
ork
pla
ces
Ind
ustr
ies
and
bus
ines
ses
and
NSW
G
over
nmen
t ag
enci
es im
ple
men
t ag
reed
WH
S in
itia
tive
s an
d a
dop
t hi
gh-
leve
l co
ntro
ls
Saf
eWo
rk n
eed
s to
p
rovi
de
rig
ht t
ouc
h re
gul
ato
ry a
pp
roac
h so
th
at c
usto
mer
s ca
n b
e co
nfid
ent
that
we
will
p
rote
ct w
ork
ers,
tha
t w
e w
ill u
se d
ata
and
insi
ght
s to
mak
e d
ecis
ions
and
p
rovi
de
inno
vati
ve
serv
ices
to
mee
t th
e ne
eds
of
bus
ines
s
Eff
ecti
ve d
eliv
ery
of
Act
ion
Are
a 3.
Bui
ld e
xem
pla
rre
gul
ator
y se
rvic
es
NSW
wo
rkp
lace
s an
d
asso
ciat
ed s
take
hold
ers,
N
SW c
om
mun
ity,
pea
k,
bo
die
s, a
sso
ciat
ions
, co
mm
unit
y le
ader
s, P
CB
Us
and
wo
rker
s S
afeW
ork
sta
ff
Saf
eWo
rk is
re
cog
nise
d a
s o
pen
an
d t
rans
par
ent;
d
rive
n by
evi
den
ce;
fost
erin
g h
ealt
h an
d
safe
ty in
nova
tio
nS
afeW
ork
is s
een
as a
tr
uste
d a
nd in
fluen
tial
re
gul
ato
r N
SW is
re
cog
nise
d a
s th
e b
est
pla
ce t
o s
tart
and
ru
n b
usin
ess
PC
BU
s ac
cess
Saf
eWo
rk’s
pro
duc
ts
and
ser
vice
s, s
atis
fied
wit
h th
ese
PC
BU
s re
view
and
imp
rove
the
ir
wo
rk p
ract
ices
, sys
tem
s an
d c
ont
rols
in
line
wit
h S
afeW
ork
’s r
egul
ato
ry,
advi
sory
and
enf
orc
men
t ap
pro
ach
and
late
st e
vid
ence
/po
licy/
stan
dar
ds
PC
BU
s/ke
y st
akeh
old
ers/
par
tner
s w
ork
colla
bor
ativ
ely
wit
h S
afeW
ork
to
imp
lem
ent
inno
vati
ve s
olu
tio
nsP
CB
Us
have
incr
ease
d c
apac
ity
to
man
age/
elim
inat
e W
HS
ris
ksS
afeW
ork
ser
vice
s, a
nd r
egul
ato
ry
reg
ime
adap
ted
acc
ord
ing
to
ev
iden
ce f
rom
dat
a sy
stem
s
Ext
erna
l fac
tors
imp
acti
ng o
n w
ork
hea
lth
and
saf
ety
(so
cial
, eco
nom
ic, n
ew t
echn
olo
gie
s, s
cien
tifi
c ad
vanc
es, n
atio
nal p
olic
y)
WHS ROADMAP EVALUATION FRAMEWORK 19
Fig
ure
2: L
og
ic m
od
el f
or
Act
ion
Are
a 1:
Em
bed
th
e h
ealt
h an
d s
afet
y la
nd
scap
e in
NS
W w
ork
pla
ces
Un
der
Act
ion
Are
a 1,
the
pro
ble
m b
ein
g a
dd
ress
ed is
th
at b
usi
nes
ses
nee
d a
ssis
tan
ce t
o e
mb
ed t
he
safe
ty la
nd
scap
e.
Inp
uts
Act
ivit
ies
Rea
chSh
ort
ter
m o
utco
mes
Med
ium
ter
m o
utco
me
Long
ter
m o
utco
mes
Tim
e, S
taff
, P
artn
ers,
B
ud
get
Un
der
take
tar
get
ed
pro
du
ctiv
e en
gag
emen
t an
d c
olla
bo
rati
on
wit
h
stak
eho
lder
s.
Pea
k in
du
stry
bo
die
s,
asso
ciat
ion
s an
d
com
mu
nit
y le
ader
s.
• Ta
rget
ed s
take
ho
lder
s ar
e aw
are
of
WH
S is
sues
/ac
cou
nta
bili
ties
.
• In
crea
sed
will
ing
nes
s to
ta
ke a
ctio
n.
• Ta
rget
ed s
take
ho
lder
s id
enti
fy w
ays
to im
pro
ve
WH
S in
th
eir
ind
ust
ries
/w
ork
pla
ces.
• Ta
rget
ed s
take
ho
lder
s ad
voca
te f
or
imp
rove
d
WH
S w
ith
in t
hei
r o
wn
se
cto
rs.
• In
du
stri
es p
roac
tive
ly
imp
lem
ent
sect
or
wid
e W
HS
init
iati
ves
to r
edu
ce
wo
rker
s’ e
xpo
sure
to
ris
ks.
• N
SW
wo
rkp
lace
s h
ave
hig
h s
afet
y st
and
ard
s.
• F
ewer
fat
alit
ies,
ser
iou
s in
juri
es, i
llnes
ses
and
m
usc
ulo
skel
etal
dis
ord
ers
at w
ork
.
• A
wel
l-in
form
ed N
SW
co
mm
un
ity
kno
wle
dg
eab
le
abo
ut
WH
S is
sues
.
• A
ll jo
bs
are
des
ign
ed t
o
be
safe
wit
h s
afe
wo
rk
pra
ctic
es a
nd
co
ntr
ols
.
• N
SW
wo
rkp
lace
s h
ave
hig
h s
afet
y st
and
ard
s.
• E
ffec
tive
RT
W p
ract
ices
ar
e em
bed
ded
in N
SW
w
ork
pla
ces.
Tim
e, S
taff
, P
artn
ers,
B
ud
get
Lau
nch
co
mm
un
icat
ion
s an
d a
dve
rtis
ing
ca
mp
aig
ns.
NS
W g
ener
al
com
mu
nit
y,
emp
loye
rs a
nd
w
ork
ers.
• In
crea
sed
aw
aren
ess
of
Saf
eWo
rk N
SW
bra
nd
.
• In
crea
sed
leve
ls o
f en
gag
emen
t w
ith
S
afeW
ork
.
• G
ener
al c
om
mu
nit
y,
emp
loye
rs a
nd
wo
rker
s ac
cess
pro
du
cts
and
se
rvic
es.
• In
crea
sed
co
nfi
den
ce a
nd
w
illin
gn
ess
to t
ake
acti
on
.
• E
mp
loye
rs im
pro
ve t
hei
r sy
stem
s an
d c
on
tro
ls.
• W
ork
ers
imp
rove
th
eir
wo
rk
pra
ctic
es.
Tim
e, S
taff
, P
artn
ers,
B
ud
get
Dev
elo
p t
ailo
red
pro
du
cts
and
ser
vice
s fo
r sp
ecifi
c in
du
stri
es a
nd
diff
eren
t si
zed
bu
sin
esse
s.
Targ
eted
ind
ust
ries
an
d d
iffer
ent
size
d
bu
sin
esse
s.
• W
ork
pla
ces
un
der
stan
d
the
diff
eren
t el
emen
ts o
f th
e H
&S
lan
dsc
ape.
• W
ork
pla
ces
pro
acti
vely
im
ple
men
t el
emen
ts o
f th
e H
&S
lan
dsc
ape.
• Ta
rget
ed e
mp
loye
rs a
nd
w
ork
ers
acce
ss p
rod
uct
s an
d s
ervi
ces.
Tim
e, S
taff
, P
artn
ers,
B
ud
get
Wo
rk w
ith
SIR
A, d
evel
op
an
d d
eliv
er p
rog
ram
s to
im
pro
ve R
TW
pra
ctic
es.
Targ
eted
ind
ust
ries
an
d b
usi
nes
ses.
• Ta
rget
ed b
usi
nes
ses
are
awar
e o
f R
TW
issu
es/
acco
un
tab
iliti
es.
• Ta
rget
ed b
usi
nes
ses
iden
tify
way
s to
imp
rove
R
TW
ou
tco
mes
in t
hei
r w
ork
pla
ces.
• Ta
rget
ed b
usi
nes
ses
imp
lem
ent
RT
W p
ract
ices
in
th
eir
wo
rkp
lace
s.
20 SAFEWORK NSW
3.2.1. Outcomes matrices and performance measures for Action Area 1: Embed the health and safety landscape in NSW workplaces
Table 1: Action Area 1 Inputs Outcomes Matrix
Inputs (for each activity)
Attributes of success Performance measures
Time, Staff, Partners, Budget
Time, staff, budget are applied to the program as planned, for each year.
Proportion of planned time, staff, partners, budget applied, for each year.
Proportion within budget.
Appropriate partners identified. Stakeholders agree all relevant partners identified.
Table 2: Activities, reach, short term outcomes and medium term outcomes
Outcome level Outcomes and attributes of success Performance measures
Undertake targeted productive engagement and collaboration with stakeholders
Activities Priority peak industry bodies, associations and community leaders effectively engaged in collaboration.
Effective collaborative mechanisms in place allow all interest groups to be involved.
Number, type of stakeholders who agree to collaborate, compared with target.
Feedback on effectiveness of engagement strategies.
Reach All priority peak industry bodies, associations and community leaders are effectively reached.
Number, type of stakeholders who collaborate on specific issues.
Areas of collaboration.
Short term outcomes
Targeted priority peak industry bodies, associations and community leaders are:
• aware of WHS issues and accountabilities for these
• report an increased willingness to take action
• identify ways to improve WHS in their industries.
Number of priority peak industry bodies, associations and community leaders who can identify WHS issues and accountabilities relevant to their industry.
Number, type of stakeholder who report they are willing to act on WHS issues.
List of agreed identified actions to improve WHS.
WHS ROADMAP EVALUATION FRAMEWORK 21
Outcome level Outcomes and attributes of success Performance measures
Medium term outcomes
Targeted stakeholders advocate for improved WHS within their own sectors, demonstrate leadership using industry forums and levers.
Industries proactively implement sector wide WHS initiatives to reduce workers’ exposure to risks, commit resources.
Number of priority peak industry bodies, associations and community leaders who advocate for improved WHS, type of advocacy.
Evidence of industry driving change (eg – lobbying, audits, provision of information and training).
Kinds of actions being advocated (eg – changes in equipment design, complaints processes).
Number, type of initiatives by sector.
Feedback on barriers to implementation.
Launch communication and advertising campaigns
Activities Campaigns and advertising designed.
Messages developed and tested for audience appeal.
Number, type of campaigns, audience.
Extent each campaign is implemented, compared with plan.
Reach Target audience/s aware of campaign messages and services and products
For each campaign, recall rates of messages and services and products.
Short term outcomes
Target audience/s have increased awareness of SafeWork NSW brand.
Target audience have increased levels of engagement with SafeWork.
Proportion of target groups report being aware of SafeWork NSW brand.
Numbers of target audience/s seek further information through visits to website.
Medium term outcomes
General community, employers and workers access products and services.
Increased confidence and willingness to take action.
Employers improve their systems and controls.
Workers improve their work practices.
Numbers of target audience/s access advertised services and products and ways services are accessed (business visits, phone calls).
Proportion of users of services and products satisfied, find these useful.
Number of employers in different types of industries who make changes to WHS systems and controls, kinds of changes made.
Proportion of workers who report changing work practices, nature of changes (drawn from a sample of businesses who have changed systems and controls).
Feedback on barriers to changing systems and controls and worker behaviour.
22 SAFEWORK NSW
Outcome level Outcomes and attributes of success Performance measures
Develop tailored products and services for specific industries and different sized businesses
Activities Specific industries and their WHS service and product needs identified.
Products and services developed, tailored for each industry and for different sized businesses within each industry, address the health and safety landscape.
Products and services effectively promoted to specified industries.
Number, type of industries and products and services implemented.
Extent each service and product is delivered as intended.
Match of products and services with industry needs.
Number and types of promotional activities.
Reach All targeted industries and different sized businesses are aware of new products and services and access these.
Numbers and sizes of businesses from targeted industries access new services and products.
Short term outcomes
Products and services useful, meet information needs in timely way.
Responsible staff in different size businesses understand elements of the H&S landscape and how these can be applied in their own workplace.
Proportion of responsible business staff who use services and products are satisfied, find these useful, get timely response.
Numbers of responsible staff in a sample of businesses of different sizes can describe relevant elements of the H&S landscape.
Medium term outcomes
Workplaces proactively implement elements of the H&S landscape.
Numbers of businesses of different sizes have fully implemented relevant elements H&S landscape (new policy, guidelines, systems and controls).
Feedback on barriers to implementing H&S landscape elements.
Work with SIRA, develop and deliver programs to improve RTW practices
Activities SafeWork works with SIRA on RTW programs, contributes expertise on health and safety issues for returning workers.
Processes to support effective collaboration established.
Number, target group for and focus of programs developed, compared with target.
Nature of contribution by SafeWork.
Feedback from stakeholders.
Reach All targeted industries and businesses are effectively reached.
Number, type of industries and businesses participate in RTW programs.
WHS ROADMAP EVALUATION FRAMEWORK 23
Outcome level Outcomes and attributes of success Performance measures
Short term outcomes
Targeted businesses are aware of relevant RTW issues/accountabilities.
Targeted businesses identify ways to improve RTW outcomes in their workplaces.
Proportion of businesses find support or information offered through programs useful.
Number, type of businesses report being aware of RTW issues and accountabilities, can describe issues relevant to their operating context and actions to address these.
Numbers of businesses intending to change RTW practices.
Medium term outcomes
Targeted businesses implement RTW practices in their workplaces.
Numbers of businesses that can demonstrate implementing relevant RTW practices, description of practices.
Rates of return to work and whether employment sustained after return.
Feedback on barriers to implementing RTW practices.
24 SAFEWORK NSW
Fig
ure
3: L
og
ic m
od
el f
or
Act
ion
Are
a 2:
Pri
ori
tise
sec
tors
, har
ms,
wo
rker
s an
d w
ork
pla
ces
wh
ere
the
mo
st s
ign
ifica
nt W
HS
ris
ks e
xist
Un
der
Act
ion
Are
a 2,
th
e p
rob
lem
bei
ng
ad
dre
ssed
is t
hat
hig
h r
isk
sect
ors
an
d h
arm
s n
eed
to
be
pri
ori
tise
d, s
o S
afeW
ork
nee
ds
to w
ork
wit
h in
du
stry
to
id
enti
fy a
nd
ad
dre
ss h
igh
ris
k se
cto
rs a
nd
har
ms,
pro
tect
th
e m
ost
at
risk
wo
rker
s an
d e
nsu
re t
hat
hig
h r
isk
wo
rkp
lace
s m
eet
com
plia
nce
sta
nd
ard
s.
Inp
uts
Act
ivit
ies
Rea
chSh
ort
ter
m o
utco
mes
Med
ium
ter
m o
utco
me
Long
ter
m o
utco
mes
Tim
e, S
taff
, P
artn
ers,
B
ud
get
Wo
rk w
ith
rel
evan
t st
akeh
old
ers
to a
dd
ress
h
igh
imp
act
har
ms
and
W
HS
issu
es o
f ta
rget
ed
hig
h r
isk
ind
ust
ries
an
d a
t ri
sk w
ork
ers.
Pea
k in
du
stry
b
od
ies,
ass
oci
atio
ns,
co
mm
un
ity
lead
ers,
em
plo
yers
an
d
wo
rker
s.
• Im
pro
ved
aw
aren
ess
of
hig
h im
pac
t h
arm
s an
d
WH
S is
sues
of
targ
eted
in
du
stri
es a
nd
wo
rker
s.
• In
crea
sed
co
nfi
den
ce,
cap
abili
ty a
nd
will
ing
nes
s to
tak
e ac
tio
n.
• In
du
stri
es a
nd
bu
sin
esse
s im
ple
men
t W
HS
init
iati
ves
and
ad
op
t h
igh
leve
l co
ntr
ols
.
• H
igh
ris
k in
du
stri
es a
re
sig
nifi
can
tly
safe
r an
d
hea
lth
ier.
• H
igh
imp
act
har
ms
are
elim
inat
ed o
r si
gn
ifica
ntl
y re
du
ced
.
• F
ewer
fat
alit
ies,
ser
iou
s in
juri
es, i
llnes
ses
and
m
usc
ulo
skel
etal
dis
ord
ers
at w
ork
.
• A
saf
e an
d h
ealt
hy N
SW
p
ub
lic s
ecto
r.
• S
afe
and
hea
lthy
w
ork
pla
ces.
Tim
e, S
taff
, P
artn
ers,
B
ud
get
Wo
rk w
ith
th
e N
SW
G
ove
rnm
ent
sect
or
to
dev
elo
p a
nd
del
iver
WH
S
init
iati
ves.
Man
agem
ent
and
st
aff o
f ta
rget
ed
NS
W G
ove
rnm
ent
dep
artm
ents
.
• Im
pro
ved
aw
aren
ess
of
WH
S is
sues
.
• In
crea
sed
co
nfi
den
ce,
cap
abili
ty a
nd
will
ing
nes
s to
tak
e ac
tio
n.
• N
SW
Go
vern
men
t d
epar
tmen
ts im
ple
men
t W
HS
init
iati
ves.
Tim
e, S
taff
, P
artn
ers,
B
ud
get
Wo
rk w
ith
hig
h r
isk
wo
rkp
lace
s to
imp
rove
W
HS
.
Man
agem
ent
and
st
aff o
f se
lect
ed h
igh
ri
sk w
ork
pla
ces.
• Im
pro
ved
aw
aren
ess
of
WH
S is
sues
.
• In
crea
sed
co
nfi
den
ce,
cap
abili
ty a
nd
will
ing
nes
s to
tak
e ac
tio
n.
• Ta
rget
ed b
usi
nes
ses
imp
lem
ent
WH
S in
itia
tive
s
• Ta
rget
ed b
usi
nes
ses
ado
pt
hig
h le
vel c
on
tro
ls.
WHS ROADMAP EVALUATION FRAMEWORK 25
3.2.2. Outcomes matrices and performance measures for Action Area 2: Prioritise sectors, harms, workers and workplaces where the most significant WHS risks exist
Table 3: Action Area 2 Inputs Outcomes Matrix
Inputs (for each activity)
Attributes of success Performance measures
Time, Staff, Partners, Budget
Time, staff, budget are applied to the program as planned, for each year.
Priority harm prevention projects identified.
Proportion of planned time, staff, budget applied, for each year.
Proportion within budget.
Number, type of projects selected, coverage of projects against identified risks.
Appropriate partners identified. Stakeholders agree all relevant partners identified.
Table 4: Activities, reach, short term outcomes and medium term outcomes
Outcome level Outcomes and attributes of success Performance measures
Work with relevant stakeholders to address high impact harms and WHS issues of targeted high risk industries and at risk workers
Activities Relevant stakeholders effectively engaged in collaborative projects on ways to address high risk harms.
Effective collaborative mechanisms in place allowing all interest groups to be involved.
Number, type of stakeholders who agree to collaborate, compared with targets identified in project plans.
Areas of collaboration, and which risks addressed (high impact harms being addressed, WHS issue).
Feedback on effectiveness of engagement strategies.
Projects delivered as intended.
Reach All stakeholders (priority peak industry bodies, associations, community leaders, employers and workers) are effectively reached.
Number, type of stakeholders who collaborate on specific high impact harm projects.
Nature of collaborative initiatives.
Short term outcomes
All stakeholders are more aware of high impact harms and WHS issues.
All stakeholders are more confident in their ability to address high impact harms in their industry or workplace.
All stakeholders are capable of and willing to take action.
Number of stakeholders who can identify high impact harms and WHS issues relevant to their industry.
Number, type of stakeholder who report they are willing to act on high impact harms and WHS issues, able to do so.
List of agreed identified initiatives to address high impact harms.
26 SAFEWORK NSW
Outcome level Outcomes and attributes of success Performance measures
Medium term outcomes
Industries and businesses implement agreed WHS initiatives, address high impact harms, industry-wide implementation.
Evidence of industry driving change (eg – provision of information and training, funding).
Number, type of initiatives by sector.
Evidence of reduced exposure to risk for workers in targeted industries.
Evidence of high-level controls being implemented.
Fewer complaints about WHS from workers.
Feedback on barriers to implementation
Extent of implementation by industry and sector.
Work with the NSW Government sector to develop and deliver WHS initiatives
Activities All targeted departments effectively engaged in collaboration to develop and deliver suitable WHS initiatives.
Effective collaborative mechanisms in place.
Number of departments who agree to collaborate, compared with targets
Areas of collaboration (description of initiatives)
Reach All responsible managers and staff members of targeted NSW Government departments are effectively reached.
Number of departments who collaborate on WHS initiatives, positions of staff involved.
Nature of joint initiatives.
Short term outcomes
All responsible managers and staff have improved awareness of WHS issues, increased confidence, capability and willingness to take action.
Number of responsible managers and staff members who can identify WHS issues relevant to their work.
Number of responsible managers and staff members who report they are willing to act on WHS issues, able to do so.
List of agreed identified actions to address WHS issues across departments.
Medium term outcomes
All targeted NSW Government departments implement agreed WHS initiatives, department leaders drive implementation.
Evidence of department managers driving change (eg – provision of information and training, guidelines).
Number, type of actions taken by departments.
Feedback on barriers to implementation.
Extent of implementation across targeted departments.
Evidence of reduced exposure to risk for workers in NSW Government departments
Fewer complaints about WHS from workers in Government departments.
WHS ROADMAP EVALUATION FRAMEWORK 27
Outcome level Outcomes and attributes of success Performance measures
Work with high risk workplaces to improve WHS
Activities All selected high risk workplaces’ managers and staff effectively engaged in collaborative initiatives to improve WHS.
Effective collaborative mechanisms in place.
Number of selected workplaces that agree to collaborate, compared with targets.
Areas of collaboration (description of initiatives).
Reach All relevant managers and staff in selected workplaces are effectively reached.
Number, type of staff who collaborate on initiatives to improve WHS.
Nature of initiatives.
Short term outcomes
All relevant managers and staff are more aware of how WHS issues are addressed.
All relevant managers and staff are more confident that WHS issues can be addressed in their workplace.
All relevant managers and staff are capable of and willing to take action.
Number of relevant managers and staff who can identify WHS issues relevant to their workplace.
number of relevant managers and staff who report they are willing to act on high risk WHS issues, able to do so.
List of agreed identified actions to address WHS issues.
Medium term outcomes
Targeted businesses implement agreed WHS initiatives.
Evidence of managers driving change (eg – provision of information and training, funding).
Number, type of actions taken by selected businesses.
Evidence of high-level controls being implemented.
Evidence of reduced exposure to risk for workers in targeted industries.
Fewer complaints about WHS from workers.
Feedback on barriers to implementation.
Extent of implementation by selected businesses.
28 SAFEWORK NSW
Fig
ure
4: L
og
ic m
od
el f
or
Act
ion
Are
a 3:
Bu
ild e
xem
pla
r re
gu
lato
ry s
ervi
ces
Un
der
Act
ion
Are
a 3
, th
e p
rob
lem
bei
ng
ad
dre
ssed
is t
hat
Saf
eWo
rk n
eed
s to
pro
vid
e ri
gh
t to
uch
reg
ula
tory
ap
pro
ach
so
th
at c
ust
om
ers
can
be
con
fid
ent
that
S
afeW
ork
will
en
forc
e W
HS
law
s to
pro
tect
wo
rker
s, t
hat
Saf
eWo
rk w
ill u
se d
ata
and
insi
gh
ts t
o m
ake
dec
isio
ns
and
pro
vid
e in
no
vati
ve s
ervi
ces
to m
eet
the
nee
ds
of
bu
sin
ess.
Inp
uts
Act
ivit
ies
Rea
chSh
ort
ter
m o
utco
mes
Med
ium
ter
m o
utco
me
Long
ter
m o
utco
mes
Tim
e, S
taff
, P
artn
ers,
B
ud
get
En
able
an
d e
mp
ow
er
wo
rkp
lace
s to
man
age
thei
r h
ealt
h a
nd
saf
ety.
NS
W w
ork
pla
ces
and
ass
oci
ated
st
akeh
old
ers.
• P
CB
Us
are
awar
e o
f S
afeW
ork
’s p
rod
uct
s an
d
serv
ices
.
• P
CB
Us
un
der
stan
d t
hei
r W
HS
res
po
nsi
bili
ties
.
• P
CB
Us
acce
ss S
afeW
ork
’s p
rod
ucts
an
d s
ervi
ces,
sat
isfie
d w
ith t
hes
e.
• P
CB
Us
revi
ew a
nd im
pro
ve t
hei
r w
ork
pra
ctic
es, s
yste
ms
and
co
ntro
ls.
• In
tera
ctio
ns w
ith c
usto
mer
s m
eet
exp
ecte
d s
tan
dar
ds
of
serv
ice.
• N
SW
is r
eco
gn
ised
as
th
e b
est
pla
ce
to s
tart
an
d r
un
b
usi
nes
ses.
• S
W N
SW
is s
een
as
a t
rust
ed a
nd
in
flu
enti
al r
egu
lato
r.
• S
W N
SW
is
reco
gn
ised
as
op
en
and
tra
nsp
aren
t,
and
dri
ven
by
evid
ence
.
• S
W N
SW
is
reco
gn
ised
as
fost
erin
g h
ealt
h a
nd
sa
fety
inn
ova
tio
n.
Tim
e, S
taff
, P
artn
ers,
B
ud
get
Cla
rify
an
d p
rom
ote
S
afeW
ork
NS
W’s
re
gu
lato
ry, a
dvi
sory
an
d
enfo
rcem
ent
app
roac
h.
NS
W w
ork
pla
ces
Saf
eWo
rk s
taff
.
• P
CB
Us
are
awar
e o
f S
afeW
ork
’s r
egu
lato
ry,
advi
sory
an
d e
nfo
rcem
ent
app
roac
h.
• P
CB
Us
und
erst
and
th
eir
WH
S r
esp
on
sib
iliti
es a
nd
w
hat
co
mp
lian
ce lo
oks
like
.
• P
CB
Us
acce
ss S
afeW
ork
’s p
rod
ucts
an
d s
ervi
ces,
sat
isfie
d w
ith t
hes
e.
• P
CB
Us
revi
ew a
nd im
pro
ve t
hei
r w
ork
pra
ctic
es, s
yste
ms
and
co
ntro
ls in
lin
e w
ith S
afeW
ork
’s
reg
ulat
ory
, ad
viso
ry a
nd
en
forc
emen
t ap
pro
ach
.
• In
tera
ctio
ns w
ith c
usto
mer
s m
eet
exp
ecte
d s
tan
dar
ds
of
serv
ice.
Tim
e, S
taff
, P
artn
ers,
B
ud
get
En
able
th
e u
se o
f d
ata
and
evi
den
ce t
o in
form
W
HS
po
licie
s an
d
pro
gra
ms.
NS
W w
ork
pla
ces
and
ass
oci
ated
st
akeh
old
ers.
Saf
eWo
rk s
taff
.
• P
CB
Us
are
awar
e o
f la
test
d
evel
op
men
ts/a
dva
nce
s in
WH
S.
• S
afeW
ork
use
dat
a sy
stem
s an
d e
vid
ence
to
in
form
dec
isio
n-m
akin
g.
• P
CB
Us
revi
ew a
nd im
pro
ve t
hei
r w
ork
pra
ctic
es, s
yste
ms
and
co
ntro
ls in
lin
e w
ith t
he
late
st
evid
ence
an
d s
tan
dar
ds.
• S
afeW
ork
ser
vice
s an
d r
egul
ato
ry
reg
ime
adap
ted
acc
ord
ing
to
ev
iden
ce f
rom
dat
a sy
stem
s.
Tim
e, S
taff
, P
artn
ers,
B
ud
get
Dev
elo
p a
nd
pro
mo
te
inn
ova
tive
ap
pro
ach
es t
o
wo
rk h
ealt
h a
nd
saf
ety
issu
es.
Pea
k b
od
ies,
as
soci
atio
ns,
co
mm
un
ity
lead
ers,
P
CB
Us
and
wo
rker
s.
Saf
eWo
rk s
taff
.
• P
CB
Us
and
key
st
akeh
old
ers
reco
gn
ise
the
ben
efits
of
wo
rkin
g
wit
h S
afeW
ork
to
dev
elo
p
inn
ova
tive
so
luti
on
s.
• S
afeW
ork
use
hu
man
ce
ntre
d d
esig
n p
rin
cip
les
to
dev
elo
p e
ffec
tive
so
luti
ons
.
• P
CB
Us,
key
sta
keho
lder
s an
d
par
tner
s w
ork
co
llab
ora
tive
ly w
ith
Saf
eWo
rk t
o im
ple
men
t in
nova
tive
solu
tio
ns.
• P
CB
Us
and
key
sta
keho
lder
s ha
ve
incr
ease
d c
apac
ity
to m
anag
e o
r el
imin
ate
WH
S r
isks
.
WHS ROADMAP EVALUATION FRAMEWORK 29
3.2.3. Outcomes matrices and performance measures for Action Area 3: Build exemplar regulatory services
Table 5: Action Area 3 Inputs Outcomes Matrix
Inputs (for each activity)
Attributes of success Performance measures
Time, Staff, Partners, Budget
Time, staff, budget are applied to the program as planned, for each year.
Proportion of planned time, staff, budget applied, for each year.
Proportion within budget.
Appropriate partners identified. Stakeholders agree all relevant partners identified.
Table 6: Activities, reach, short term outcomes and medium term outcomes
Outcome level Outcomes and attributes of success Performance measures
Enable and empower workplaces to manage their health and safety
Activities Products and services for workplaces/PCBUs provide clear and appropriate advice about WHS issues and how to manage these.
Products and services effectively promoted to PCBUs.
Number, type of products and services implemented.
Projects implemented as intended.
List of SafeWork information products and audience for these, gaps in information.
Extent information meets SafeWork communication protocols/guidelines.
Extent to which products and services meet the needs of customers.
Number, type of promotional activities.
Reach All workplaces and associated stakeholders are effectively reached.
Number of PCBUs who seek information about services and products, when needed.
Short term outcomes
All PCBUs are aware of products and services.
All PCBUs understand their WHS responsibilities, relevant to their business.
Number of PCBUs who can identify WHS responsibilities relevant to their business.
Number of PCBUs who are aware of products and services.
Medium term outcomes
PCBUs access products and services when needed.
PCBUs review and improve their work practices, systems and controls in response to advice received.
SafeWork staff interactions with PCBUs meet expected standards of service.
Number of PCBUs who access services and products.
Proportion of PCBUs who use services and products are satisfied, find these useful, get timely response.
Number of PCBUs review and improve work practices, systems and controls.
Number of complaints about services.
30 SAFEWORK NSW
Outcome level Outcomes and attributes of success Performance measures
Clarify and promote SafeWork NSW’s regulatory, advisory and enforcement approach
Activities Appropriate products and services for PCBUs provide clear information on SafeWork’s regulatory, advisory and enforcement approach.
Information effectively promoted to PCBUs.
Number, type of products and services implemented.
Projects implemented as intended.
Number, type of promotional activities.
Reach All NSW workplaces are effectively reached.
Number of PCBUs who seek information about services and products related to compliance and enforcement.
Short term outcomes
PCBUs are aware of SafeWork’s regulatory, advisory and enforcement approach.
PCBUs understand their WHS responsibilities, what compliance looks like in their type of business and understand penalties for non-compliance.
Relevant information developed, is clear and consistent across the agency.
Number of PCBUs who are aware of products and services related to compliance and enforcement.
Number of PCBUs who can accurately describe SafeWork’s regulatory, advisory and enforcement approach and what compliance looks like in their business.
SafeWork interactions with PCBUs.
List of SafeWork information products and audience for these, gaps in information.
Extent information meets SafeWork communication protocols/guidelines.
Medium term outcomes
PCBUs access SafeWork’s regulatory, advisory and enforcement services.
PCBUs review and improve their work practices, systems and controls in line with SafeWork’s regulatory, advisory and enforcement approach.
Number of PCBUs who access services and products related to compliance and enforcement.
Proportion of PCBUs who use services and products are satisfied, find these useful, get timely response.
Number of PCBUs who review and improve work practices, systems and controls, extent these are in line with regulatory, advisory and enforcement approach.
WHS ROADMAP EVALUATION FRAMEWORK 31
Outcome level Outcomes and attributes of success Performance measures
Enable the use of data and evidence to inform WHS policies and programs
Activities SafeWork makes data and evidence available to PCBUs about WHS developments and advances.
Evidence is translated and easy to understand for different types of industries and businesses, readily accessible.
SafeWork establishes data systems to collect evidence and support risk-based regulatory approach.
Number and topics of papers published/presented/advice provided about WHS developments and advances (amount of dissemination).
Extent evidence is produced and disseminated compared to plan.
Data systems functioning according to specifications and requirements, gaps in functions.
Reach All NSW workplaces and associated stakeholders are effectively reached.
Number of PCBUs who report published information is useful and relevant.
Number of PCBUs who seek and find information about data and evidence on WHS developments and advances.
Data analytics meet SafeWork information needs.
Short term outcomes
PCBUs are aware of latest developments/advances in WHS and how these apply to their business.
SafeWork’s managers and executive seek out and use evidence to inform decision-making.
Number of PCBUs who are aware of WHS developments and advances.
Number of PCBUs who can accurately describe how the evidence can be applied to their business.
Examples of how data has informed decision-making.
Medium term outcomes
PCBUs review and improve their work practices, systems and controls in line with latest evidence about effective WHS practices, systems and controls.
SafeWork systems, services and policies adapted according to evidence base.
Number of PCBUs who access information products about latest WHS developments and advances.
Proportion of PCBUs who access information or advice are satisfied, find these useful.
Number of PCBUs who review and improve work practices, systems and controls, extent these are in line with latest WHS developments and advances.
Description of adaptations made to SafeWork systems, services and policies, rationale for change.
32 SAFEWORK NSW
Outcome level Outcomes and attributes of success Performance measures
Develop and promote innovative approaches to work health and safety issues
Activities Peak bodies, associations, community leaders, PCBUs and workers effectively engaged in collaboration.
Effective collaborative mechanisms in place allow all interest groups to be involved.
SafeWork actively promotes innovation internally and within the business community.
SafeWork use human centred design principles to develop effective solutions.
Number, type of stakeholders who agree to collaborate, compared with target.
Feedback on effectiveness of engagement strategies.
Change requests, number of innovative projects.
Reach Peak bodies, associations, community leaders, PCBUs and workers are effectively reached, agree on issues to be addressed.
Type of joint initiatives and innovations agreed, WHS issues being addressed.
Short term outcomes
PCBUs and key stakeholders recognise the benefits of working with SafeWork to develop innovative solutions.
Number of stakeholders who agree that working with SafeWork is beneficial, believe that joint working can address WHS issues.
Medium term outcomes
PCBUs, key stakeholders and partners work collaboratively with SafeWork to implement innovative solutions.
PCBUs and key stakeholders have increased capacity to manage or eliminate WHS risks.
Number, type of stakeholders who collaborate on specific issues and innovative approaches.
Number of PCBUs who implement innovations.
Description of innovation and how it impacts on business capacity to manage and eliminate risks.
WHS ROADMAP EVALUATION FRAMEWORK 33
3.2.4. Outcomes matrix for long term outcomes
Long term outcomes are separated because the achievement of these depends on the combined successful implementation of all three action areas and on actions taken by industry and government partners, national efforts to address work health and safety and on external social influences.
Table 7: Outcomes matrix for Roadmap long term outcomes
Outcomes Performance measures
NSW workplaces have high safety standards
Proportion of workplaces that meet safety standards.
Fewer fatalities, serious injuries, illnesses and musculoskeletal disorders at work
Achievement against Fatality Target – 20% decline (1 Quarter Lag).
Number of NSW Fatalities.
Fatality Rate.
Achievement against Serious Injury and Illness Target – 30% decline (1 Quarter Lag).
Number of NSW Serious Injuries and Illnesses.
Serious Injuries and Illnesses Rate.
% to Serious Musculoskeletal Injuries and Illnesses Target – 30% decline (1 Quarter Lag).
Number of NSW Serious Musculoskeletal Injuries and Illnesses.
Serious Musculoskeletal Injuries and Illnesses Rate.
Sustained reduction in Fatalities and Serious Injury and Illness Rates.
A well-informed NSW community knowledgeable about WHS issues
Proportion of the community who understand key work health and safety issues.
Fewer adverse social and economic impacts on families.
All jobs are designed to be safe with safe work practices and controls
Number and proportion of businesses where jobs are designed with safe work practices and controls by industry category.
Number and proportion of government agencies where jobs are designed with safe work practices and controls by industry category.
Effective RTW practices are embedded in NSW workplaces
Proportion of workplaces with appropriate return to work processes that are consistently applied.
NSW recognised as the best place to run a business
Lower cost of running a business compared to baseline (and adjusted for CPI increases).
Reduced regulatory impost on business.
SafeWork seen as a trusted and influential regulator
Number of stakeholders who agree that SafeWork is a credible, trustworthy and responsive organisation.
SafeWork is recognised as open and transparent and driven by evidence
Number of stakeholders who agree that SafeWork is an open and transparent organisation, driven by evidence.
SafeWork is recognised as fostering health and safety innovation
Number of stakeholders who agree that SafeWork fosters health and safety innovation.
34 SAFEWORK NSW
3.3. Roadmap evaluation and guiding questions
Evaluation questions support reality-testing, which is finding out what is actually going on in a program and its effects. This can be compared to what was intended and hoped for as stated in the WHS Roadmap for NSW 2022. Evaluation questions also shape the focus of the evaluation efforts.
Guiding evaluation questions point towards additional information needed to answer the evaluation questions for each action area. The outcome matrices prescribe the full set of information that can potentially be collected to answer the evaluation questions, and likely sources of data. What is actually collected then depends on decisions about identified priorities, the available resources, and the feasibility and ethical safeguards for collecting these data.
Evaluation studies will address one or more of the key evaluation questions, which span the logic model (Figure 1) and the three action areas from bottom to top. Which evaluation questions you address will depend on the purpose and scope of your evaluation study. A bank of guiding questions is proposed for each evaluation question – these may be specific to action areas, and have informed the identification of outcomes matrix measures. You can choose from the bank of guiding questions or add guiding questions to clarify what specific aspects of a project you are interested in assessing.
SafeWork core services are assistance services, public awareness (excluding major media campaigns), consultative mechanisms, prevention programs, information and advice services, financial incentives, sanctions, transparency and accountability, regulatory response, authorisations, verifications and audits.
Evaluation questions
Inputs
1. Were adequate resources allocated (staff, time and budget) to research, develop and deliver initiatives under each of the Roadmap action areas and for each Directorate?
2. How consistent are project communication materials and protocols with the right touch regulatory approach?
3. Were initiatives selected based on the best evidence available?
4. Has input from partners shaped the design of initiatives and in what ways?
Activities
5. To what extent were initiatives implemented as intended?
6. How well did SafeWork manage risks and address barriers to implementation?
7. How appropriate were project activities for the task/s?
8. How appropriate were tailored products and services for engaging stakeholders and for meeting the information needs of different audiences?
9. To what extent were communication materials and protocols consistent across the organisation?
10. What unexpected or emerging factors influenced the implementation of initiatives?
11. What consultations have taken place, who participated and have changes been made to improve SafeWork services and products as a result?
WHS ROADMAP EVALUATION FRAMEWORK 35
Reach
12. How effective were the actions in reaching their target audiences? For each target audience, what was the pattern of reach (number, proportion, characteristics of those reached)?
13. What methods were more or less effective in reaching their target audiences? What success factors and barriers were identified?
Short term outcomes
14. Are SafeWork initiatives under each action area achieving their intended short term outcomes?
Guiding questions for Action Area 1: Embed the health and safety landscape in NSW workplaces
• To what extent are the general community, employers and workers who were reached by SafeWork initiatives:
a. aware of the SafeWork brand, and has the level of awareness changed?
b. engaged with SafeWork; what is the nature of the engagement and how has it changed over time?
• Which engagement strategies with small, medium and large businesses and peak groups of employers and employees have been more or less successful?
• To what extent are peak industry bodies, associations and community leaders
a. aware of their WHS issues and accountabilities and has their level of awareness changed?
b. able to identify ways to improve work health and safety in their industries?
• To what extent do businesses of all sizes better understand the relevant elements of the health and safety landscape?
• Are targeted businesses more aware of return to work issues and accountabilities and able to identify ways to improve their processes?
Guiding questions for Action Area 2: Prioritise sectors, harms, workers and workplaces where the most significant WHS risks exist
• What has been done to elevate work health and safety in high risk workplaces?
• To what extent are targeted industries (peaks, employers and workers) aware of high impact harms and health and safety risks and has their level of awareness changed?
• To what extent are managers in NSW Government agencies aware of WHS issues and has their level of awareness changed?
• To what extent are targeted industries, management and staff in selected high risk workplaces and managers in NSW Government agencies more confident, capable and willing to act on health and safety issues in the workplace?
• To what extent are targeted industries aware of relevant SafeWork products and services?
36 SAFEWORK NSW
Guiding questions for Action Area 3: Build exemplar regulatory services
• What has been done to elevate work health and safety in workplaces?
• Is the compliance regime being consistently delivered under the legislation?
• What kinds of impacts have compliance activities had on industry attitudes to health and safety?
• What actions have non-compliant businesses taken to address work health and safety issues identified by inspectors?
• What has been done to elevate work health and safety in workplaces and how does this translate to contracts that are won?
• To what extent are persons conducting businesses and undertakings:
a. aware of relevant SafeWork products and services and has their level of awareness changed?
b. aware of SafeWork’s regulatory, advisory and enforcement approach?
c. understanding of their work health and safety responsibilities and what compliance looks like?
d. aware of SafeWork products and services?
e. aware of latest developments and advances in work health and safety?
• Do persons conducting businesses and undertakings, peak bodies and associations, workers and community leaders recognise the benefits of working with SafeWork to address work health and safety issues?
15. What factors influenced the achievement of short term outcomes?
16. What changes to services, products and projects are needed to better achieve the intended outcomes, and can lessons be applied to other work?
Medium term outcomes
17. Are SafeWork initiatives under each action area achieving their intended intermediate outcomes?
Guiding questions for Action Area 1: Embed the health and safety landscape in NSW workplaces
• How many businesses of all sizes have implemented or upgraded work health and safety landscapes as a result of SafeWork NSW’s intervention and improved their safety systems and controls; what kinds of businesses and industries?
• How many businesses have implemented a strategy to improve or work towards a mentally healthy workplace?
• How many businesses have implemented a strategy to reduce the risk of musculoskeletal disorders and what types of controls are in place?
• In what ways have Roadmap partners (peak industry bodies, associations and community leaders) advocated to encourage workplaces to improve health and safety and reduce risks?
• To what extent are general members of the community, employers and workers more confident, capable and willing to act on health and safety issues in the workplace?
• To what extent have workers changed their practices to work in a healthier and safer way?
• How many businesses of all sizes have a return to work plan in place as a result of SafeWork’s intervention?
• What products and services have the community and workers accessed; who accessed which services/products and how many?
WHS ROADMAP EVALUATION FRAMEWORK 37
Guiding questions for Action Area 2: Prioritise sectors, harms, workers and workplaces where the most significant WHS risks exist
• Has targeting workplaces in high risk sectors and sectors where workers are at risk of high impact harms improved compliance and identification of safety risks in these industries?
• What kinds of actions to address health and safety issues have been taken by targeted industries, high risk workplaces and government agencies?
• In the chemicals area, what behaviors and practices have changed in targeted industries to targeted controls? Have these changes been sustained? Do industries have the capability to sustain these changes going forward?
• To what extent has workers’ exposure to hazards been reduced in workplaces targeted by SafeWork NSW?
• How many targeted employers have improved work practices and changed behaviours to better protect workers’ health and safety and specifically, those businesses in sectors where workers at high risk of harm and/or in high risk industries or government agencies?
Guiding questions for Action Area 3: Build exemplar regulatory services
• What products and services have persons conducting businesses and undertakings accessed; who accessed which services/products and how many?
• Do compliance and enforcement activities influence behaviour and what activities are most effective?
• What kinds of actions to identify and then address health and safety issues have been taken by persons conducting businesses and undertakings? Are the actions in line with the latest evidence?
• Are businesses of all sizes better equipped and more able to manage health and safety issues?
• In what areas of health and safety has SafeWork collaborated with peak bodies, associations and community leaders?
• What kinds of health and safety innovations have been introduced as a result of collaboration with industry?
• Has the Roadmap delivered a level playing field in terms of competitiveness within industries?
18. Were there any unintended consequences with respect to the successful implementation of the three actions areas for the community, industry, employees and workers?
19. What external factors influenced the achievement of medium term outcomes?
20. Have the lessons learnt been applied across the organisation to better achieve outcomes?
Long term outcomes
21. What impact has the Roadmap had on trends in serious workplace injuries, illnesses, musculoskeletal disorders and fatalities in different sectors?
22. What external factors are influencing the observed trends in serious workplace injuries, illnesses, musculoskeletal disorders and fatalities in different sectors?
23. In what ways has the Roadmap contributed to achieving SafeWork’s vision of workers in NSW leading healthy, safe and productive working lives?
38 SAFEWORK NSW
Guiding questions for Action Area 1: Embed the health and safety landscape in NSW workplaces
• What impact has the Roadmap had on how SafeWork is perceived by industry, workers and the community?
• What impact has the Roadmap had on the NSW community’s understanding about work health and safety issues?
• What impact has the Roadmap had on physical and mental environments in the workplace and on supply chains?
• What impact has the Roadmap had on making safe return to work practices part of business as usual in NSW workplaces?
Guiding questions for Action Area 2: Prioritise sectors, harms, workers and workplaces where the most significant WHS risks exist
• What impact has the Roadmap had on health and safety standards in government and the private sectors?
• What impact has the Roadmap had on health and safety in high risk industries?
• What impact has the Roadmap had on the incidence and prevalence of high impact harms?
Guiding questions for Action Area 3: Build exemplar regulatory services
• What impact has the Roadmap had on business perceptions about NSW as the best place to start and run a business?
• What impact has the Roadmap had on perceptions about SafeWork’s reputation as an agency that fosters health and safety innovation?
24. What external factors are influencing the observed trends in action area policy outcomes?
WHS ROADMAP EVALUATION FRAMEWORK 39
4. Performance monitoringSafeWork has performance monitoring systems in place and is further refining these to collect performance information.
4.1. About performance monitoringPerformance monitoring system is the provision of outcome information about whether or not a project/program is on track, specifically about how much has been done to what quality and sometimes, the short term outcomes achieved. Performance monitoring:
• is often a part of evaluation, but monitoring by itself is not evaluation and is less in-depth than evaluation
• is ongoing while evaluation is typically periodic
• mainly uses data captured as part of delivering services
• informs quality assurance activities, and
• is used to manage programs/businesses on an ongoing basis.
A good monitoring system provides the required monitoring as performance indicators and also provides those who submit data with reports that explain how their information has been used to motivate them to continue to provide good quality data.
What monitoring systems are
Feature Purpose
Logic Needs to be clear about what success looks like, used to identify indicators.
Measurement Outlines what is feasible to measure and source of data.
Reporting Communicates progress and achievements (eg – dashboard reports).
Improvement Indicates how the information will be used to improve things (quality assurance activities).
Accountability Identifies appropriate checks and balances (eg – data quality checks).
40 SAFEWORK NSW
How monitoring relates to program logic and evaluation
Program logic Domain Potential evaluation methods
Long term outcomes
Evaluation/research
ABS Work-related Injuries Survey
Traumatic Fatalities Database
Workers Compensation Claims Database
Longitudinal studies
Medium term outcomes
Evaluation Post surveys, interviews and observation, case studies
Longitudinal studies
Short term outcomes
Monitoring Lead indicator data collected via surveys and web analytics to assess reach, administrative databases
Reach Monitoring Administrative databases
Surveys
Interviews
Domain of monitoring
Activities Monitoring Document reviews
Interviews
Inputs Evaluation Review of initiative and project plans, other documents
4.2. Collection of performance metricsThe evaluation framework should be complemented by a monitoring plan to progressively collect performance data that will be needed for the evaluation activities.
Organisational-wide and directorate-level performance or lead indicators and metrics are being developed in 2017 and systems for measuring these are being identified or existing systems adapted. Six lead indicators and metrics will track the achievement of immediate outcomes that will be shared externally, quarterly, to track the progress of the Roadmap. These metrics have been included where known in the outcomes matrices in chapter 3.
4.2.1. Reporting performance metrics
SafeWork is designing a balanced score card to report performance information to potential internal and external users of the information.
4.3. AccountabilitySafeWork is currently reviewing and refining data quality checking processes.
WHS ROADMAP EVALUATION FRAMEWORK 41
5. Evaluation governance, evaluation planning and advice on methods
This section suggests governance structures for evaluation activity, a decision support tool and describes the evaluation planning cycle.
5.1. Governance and management of evaluation studiesStrong and effective governance and management structures for the evaluation of Roadmap initiatives are necessary to ensure that appropriate evaluation studies are conducted and stakeholders are engaged in the evaluation. Governance structures also provide a mechanism to inform program improvement and policy decisions. For the Roadmap, it is particularly important to have governance structures in place that facilitate decision-making and allow advice to be sourced when partners are involved in an initiative, and when an external team is conducting the evaluation. One overarching principle is that the governance arrangements for the evaluation of the Roadmap should be aligned with the governance structures for the delivery of the Roadmap.
The Executive (or relevant governance committee) will have oversight for evaluation and for decision-making about funding of individual evaluation studies. The Committee will be guided by the decision support tool shown in section 5.4. Business Performance will provide advice to the relevant governance committee (as requested), manage the commissioning of external evaluation studies and be responsible for ensuring that evaluation activities dovetail with the regular monitoring of Roadmap outputs.
Role and functions of evaluation steering committees
A steering committee is appropriate when a new program of work is planned and where this work involves multiple sectors and stakeholders; and where decisions about program improvement and policy are expected to be influenced directly by the findings of the evaluation.
The relevant governances committee’s role in regard to evaluation governance could encompass:
• supporting the implementation of robust and credible evaluation studies across each action area
• making decisions about which initiatives should be evaluated, based on the application of the decision support tool and on advice from Business Performance (as requested)
• making decisions about funding amounts and mix for evaluation studies
• making decisions about the procurement of external evaluation services
• approving detailed evaluation plans for Tier 3 and Tier 4 evaluations
• accepting evaluation reports
• identifying key policy and program implications
• supporting the dissemination of findings to key decision makers, operational officers.
Role and functions of evaluation advisory or working groups
Evaluation advisory groups are time limited and formed to provide specific advice for an evaluation study. Having an evaluation advisory or working group is important where there are multiple agencies and stakeholders involved and when sector specific advice will be needed to refine data collection methods and engage stakeholders in the evaluation. Advisory groups are also useful in these circumstances because they are able to provide in-depth knowledge about the program or policy being implemented. During the implementation of an evaluation the usual roles and functions of evaluation advisory groups encompass providing advice about:
• planning and implementing specific evaluation activities
• how, and by whom, issues impacting on the implementation of the evaluation should be raised, escalated and resolved
42 SAFEWORK NSW
• where or from whom to get sector specific advice where the expertise is not available from within the advisory group
• how to communicate with stakeholders about the evaluation and protocols for doing so
• the accuracy of findings, evaluation reports and other products
• how to interpret and contextualise findings
• how to disseminate the evaluation findings for program or policy improvement and accountability.
Recommendation for the Roadmap: Evaluation advisory groups are established for Tier 3 and Tier 4 level evaluation studies (section 5.4) with terms of reference that clearly communicate the role of the group. These advisory groups would be drawn from and sit under the relevant program governance committees. Additional members could include relevant SafeWork operational level staff and representatives of key peak bodies, associations and community leaders. Alternatively, input into evaluation design and considerations of the implications of findings be sought from external stakeholders via existing engagement structures.
5.2. Directorate-level annual evaluation planningEach Directorate should plan evaluation studies as part of the annual cycle of business planning. Such planning would involve identifying and proposing evaluation studies for the following year and how the results will be used to inform decision-making about service improvement and expansion and/or continuation of initiatives.
Directorate annual evaluation planning should also identify which external partners will be involved in the planned evaluations and the manner of that involvement. Ways that external partners could get involved include, as:
• members of advisory groups for evaluation studies and
• key informants about the implementation of initiatives.
5.3. Individual project evaluation plansAt the individual project level, all registered projects plans should be inclusive of evaluation; that is the project plan should identify the evaluation tier, methods and oversight.
5.4. Decision support toolIt is not feasible, cost effective or appropriate to fully evaluate all initiatives under the Roadmap. In recognition of this, a decision tool has been developed, ‘Roadmap Evaluation Priority Levels’ (Table 8). This tool is intended to assist SafeWork to make decisions about what kind of evaluation is needed and how to prioritise budget resources. The tool describes four tiers of evaluation with 1 being a very simple program or change in service and 4 representing a substantial investment in an action area under the Roadmap. The approach to evaluation is described for each level. The four tiers are a guide to assist SafeWork directorates to scale evaluation planning and for SafeWork executive to allocate resources for evaluation.
Each Tier 2, 3 and 4 evaluation study must have an evaluation and data collection plan that sets the purpose, stages, methods and timeline for the evaluation, and who will be responsible for specific tasks. Project managers will develop evaluation plans in conjunction with or using guidance from Business Performance staff.
WHS ROADMAP EVALUATION FRAMEWORK 43
Tab
le 8
: Th
e R
oad
map
Eva
luat
ion
Pri
ori
ty L
evel
s5
Tier
Typ
e o
f p
roje
ctE
valu
atio
n ap
pro
ach
Ove
rsig
htE
valu
atio
n au
die
nce
and
co
mm
unic
atio
n
1M
ino
r to
mo
der
ate
enh
ance
men
ts
to c
ore
ser
vice
wit
h a
fo
cus
on
se
rvic
e im
pro
vem
ent*
.
No
or
very
sm
all b
ud
get
ary
inve
stm
ent
in c
han
ge.
Inte
rnal
sta
keh
old
ers
on
ly.
No
str
ateg
ic o
r ac
cou
nta
bili
ty
imp
licat
ion
s u
nd
er t
he
Ro
adm
ap.
• Id
enti
fied
in a
nn
ual
bu
sin
ess
pla
n a
nd
in a
pp
rove
d p
roje
ct p
lan
s.
• P
roce
ss e
valu
atio
n, l
imit
ed s
cop
e.
• R
eso
urc
ed f
rom
wit
hin
bu
sin
ess
bu
dg
et.
Pro
gra
m m
anag
er a
nd
/o
r ev
alu
ato
r.R
epo
rtin
g w
ith
in
Dir
ecto
rate
.
2S
ub
stan
tive
ch
ang
e to
co
re s
ervi
ce
un
der
th
e R
oad
map
or
pro
ject
.
New
ap
pro
ach
or
pilo
t in
itia
tive
.
Rel
ativ
ely
low
inve
stm
ent
nee
ded
fo
r in
itia
tive
to
be
del
iver
ed.
Ext
ern
al s
take
ho
lder
s co
nsu
lted
.
Co
ntr
ibu
tes
to a
per
form
ance
m
easu
re.
• Id
enti
fied
in a
nn
ual
pla
n a
nd
in a
pp
rove
d p
roje
ct p
lan
s.
• E
valu
atio
n f
ocu
s: p
roce
ss, r
each
an
d im
med
iate
ou
tco
mes
.
• R
eso
urc
ed f
rom
wit
hin
th
e b
usi
nes
s b
ud
get
.
• P
roce
ss e
valu
atio
n fi
nd
ing
s av
aila
ble
wit
hin
12
mo
nth
s an
d s
tud
y is
co
mp
lete
d w
ith
in t
wo
-yea
r ev
alu
atio
n c
ycle
.
Rel
evan
t D
irec
tor.
Co
nsi
der
ad
vice
fro
m
Bu
sin
ess
Per
form
ance
o
n e
valu
atio
n d
esig
n.
Rep
ort
ing
wit
hin
D
irec
tora
te.
Rep
ort
may
be
circ
ula
ted
to
inte
rnal
an
d r
elev
ant
exte
rnal
st
akeh
old
ers.
3S
trat
egic
ally
sig
nifi
can
t p
roje
cts.
Mo
der
ate
inve
stm
ent.
Ext
ern
al s
take
ho
lder
s co
nsu
lted
or
par
tner
s in
th
e in
itia
tive
.
• S
tatu
s id
enti
fied
in a
nn
ual
eva
luat
ion
pla
n a
nd
in a
pp
rove
d
pro
ject
pla
ns.
• P
roje
ct p
lan
incl
usi
ve o
f ev
alu
atio
n p
lan
; sta
ged
imp
lem
enta
tio
n.
• E
valu
atio
n f
ocu
s o
n p
roce
ss, r
each
, im
med
iate
ou
tco
mes
an
d
inte
rmed
iate
ou
tco
mes
.
• M
eth
od
s m
ay in
clu
de
colle
ctio
n o
f b
asel
ine
dat
a an
d o
utc
om
es
ove
r ti
me.
• F
ind
ing
s av
aila
ble
aft
er e
ach
sta
ge
of
eval
uat
ion
.
• A
n a
lloca
ted
bu
dg
et/r
eso
urc
es t
o c
om
ple
te e
valu
atio
n.
• C
on
sid
er in
dep
end
ent
eval
uat
or
or
pee
r re
view
.
• F
ull
eval
uat
ion
rep
ort
.
Saf
eWo
rk e
xecu
tive
or
rele
van
t G
ove
rnan
ce
Co
mm
itte
e m
ust
en
do
rse
eval
uat
ion
pla
n.
Pro
gra
m g
ove
rnan
ce
com
mit
tee
advi
sory
ro
le.
See
k ad
vice
fro
m
Bu
sin
ess
Per
form
ance
o
n e
valu
atio
n d
esig
n.
Rep
ort
pu
blic
ly
avai
lab
le.
5 T
iers
dev
elo
pe
d f
rom
NS
W G
ove
rnm
ent
Eva
luat
ion
Fra
mew
ork
, 20
16.
44 SAFEWORK NSW
Tier
Typ
e o
f p
roje
ctE
valu
atio
n ap
pro
ach
Ove
rsig
htE
valu
atio
n au
die
nce
and
co
mm
unic
atio
n
4S
ign
ifica
nt
inve
stm
ent
in a
ctio
n a
rea
(rel
ativ
e to
SW
NS
W).
Dir
ect
invo
lvem
ent
of
par
tner
s b
oth
g
ove
rnm
ent
and
ind
ust
ry.
Hig
h p
rofi
le.
Fin
anci
al o
r o
rgan
isat
ion
al
imp
licat
ion
s.
• C
om
pre
hen
sive
eva
luat
ion
str
ateg
y, f
ocu
s o
n o
utc
om
es a
nd
im
pac
ts.
• M
eth
od
s m
ust
incl
ud
e co
llect
ion
of
bas
elin
e d
ata
and
ou
tco
mes
o
ver
tim
e.
• S
tag
ed d
eliv
ery
of
fin
din
gs.
• In
dep
end
ent
eval
uat
or
or
pee
r re
view
.
• A
n a
lloca
ted
bu
dg
et/r
eso
urc
es t
o c
om
ple
te e
valu
atio
n.
• F
ull
eval
uat
ion
rep
ort
.
Saf
eWo
rk e
xecu
tive
or
rele
van
t G
ove
rnan
ce
Co
mm
itte
e m
ust
en
do
rse
eval
uat
ion
pla
n.
Pro
gra
m g
ove
rnan
ce
com
mit
tee
advi
sory
ro
le.
Rep
ort
pu
blic
ly
avai
lab
le a
nd
sen
t to
M
inis
ter.
No
tes:
*C
ore
ser
vice
s ar
e as
sist
ance
ser
vice
s, p
ub
lic a
war
enes
s (e
xclu
din
g m
ajo
r m
edia
cam
pai
gn
s), c
on
sult
ativ
e m
ech
anis
ms,
pre
ven
tio
n p
rog
ram
s, in
form
atio
n
and
ad
vice
ser
vice
s, fi
nan
cial
ince
nti
ves,
san
ctio
ns,
tra
nsp
aren
cy a
nd
acc
ou
nta
bili
ty, r
egu
lato
ry r
esp
on
se, a
uth
ori
sati
on
s, v
erifi
cati
on
s an
d a
ud
its.
WHS ROADMAP EVALUATION FRAMEWORK 45
Relationship to SafeWork’s Project Prioritisation Tool
To assist in decision-making during the business planning phase as to which projects shall proceed, SafeWork has developed a Project Prioritisation Tool. The tool comprises six criteria that are applied to each potential project to generate a rating against each criteria and an overall rating score. The criteria are harm prevention (frequency of incidents); harm prevention (injury severity); strategic alignment; economic impact; customer reach; and operational excellence. The projects will then be given a ranking that corresponds to the relative scores of all projects.
The Project Prioritisation Tool and Evaluation Priority Tiers have different purposes. The Project Prioritisation Tool is used to select from all potential projects. The Evaluation Priority Tiers consider the evaluability of a project, the nature of the service innovation or project and the size of investment in innovation.
5.5. Evaluation checklistThe checklist below (Figure 5) has been designed to assist SafeWork staff members when planning evaluations.
Figure 5: Evaluation checklist
Project or initiative name
Purpose statement of project/program
Why do you want to do an evaluation
What questions are you seeking answers to
Proposed methodology and data sources, consider feasibility and cost
Proposed target group
Source of budget
Proposed budget
Involvement of external partners and who
46 SAFEWORK NSW
5.6. Methodological approachesAny particular approach to an evaluation will depend on what is considered credible, feasible, useful and possible, given the nature of the initiative being evaluated, the information needs of the project team, senior managers and corporate, and the available skills and resources. The methodological approach chosen will coincide with the evaluation tier the study has been placed in.
Being explicit about the reason/s for evaluating your initiative helps decide what data to collect and when. What are you seeking to do?
• Monitor how much is being done and how well
• Improve your service/product/project
• Assess outcomes/benefits/impacts.
One of the biggest risks in impact evaluation is getting a precise but inaccurate answer about the impact of an initiative or project. This often leads to decisions based on a false sense of certainty where a less ambitious approach may lead to more accurate answers about a more limited range of impacts. This risk is highest when only one method is used, even if this method is considered robust. Mixed methods are the best way to reduce this risk. For example, professionally gathered stories can assist in explanations of what hard data means and help understand the contexts in which a particular initiative was effective and through what mechanisms, such that past successes may be replicated in future contexts.
For many Roadmap evaluations, the use of mixed methods will be suitable, drawing on both qualitative and quantitative data. Evidence from different sources can be triangulated to support or qualify findings. Evaluation plans should ensure the use of the most robust and credible evidence that is available.
Appropriate and feasible data collection methods should be used to collect monitoring and evaluation data that is not available through existing sources. This level of detail, along with sampling frames and timelines, will need to be developed in the evaluation plans for each project, based on the program logic developed for that project.
The table on page 47 provides an overview of key data collection methods that are commonly used in a mixed methods evaluation.
WHS ROADMAP EVALUATION FRAMEWORK 47
Tab
le 9
: Co
mm
on
ly u
sed
met
ho
ds
in e
valu
atio
n6
Met
hod
Stre
ngth
sW
eakn
esse
sC
ons
ider
atio
ns
Sur
vey
Can
gen
erat
e a
larg
e am
oun
t o
f q
uant
itativ
e an
d
qua
litat
ive
dat
a.D
oes
no
t al
low
fo
r in
tera
ctio
n o
r p
rob
ing
of
resp
ons
es. S
urve
ys a
re b
eco
min
g u
biq
uito
us
and
may
no
t b
e su
ffici
ently
sal
ient
to
ob
tain
su
ffici
ent
resp
ons
e ra
tes
to b
e us
eful
. S
elf-
rep
ort
ed c
hang
es.
Mus
t b
e d
evel
op
ed a
nd t
este
d t
o
ensu
re q
uest
ions
are
co
mp
rehe
nsiv
e an
d u
nam
big
uous
and
sam
ple
is
rep
rese
ntat
ive
of
the
po
pul
atio
n
of
inte
rest
.
Fo
cus
gro
ups
Allo
w f
or
exp
lora
tion
of
key
issu
es w
hen
they
are
re
lativ
ely
com
ple
x o
r no
t to
p-o
f-m
ind
fo
r re
spo
nden
ts.
Pro
vid
e re
spo
nden
ts w
ith o
pp
ort
unit
y fo
r le
arni
ng f
rom
o
ther
s fir
st-h
and
.
Diffi
cult
to a
naly
se in
div
idua
l res
po
nses
and
no
t su
ited
to
qua
ntita
tive
anal
ysis
.R
equi
re a
ski
lled
fac
ilita
tor
and
the
ab
ility
to
bri
ng p
artic
ipan
ts t
og
ethe
r.
Wo
rksh
op
sC
an d
raw
on
colle
ctiv
e kn
owle
dg
e o
f d
ecis
ion
mak
ers
to in
terp
ret
evid
ence
and
gen
erat
e re
leva
nt a
nd u
sefu
l im
plic
atio
ns a
nd fi
ndin
gs.
The
util
isat
ion
focu
s m
eans
the
y m
ay n
ot
wel
l su
ited
fo
r p
rim
ary
dat
a co
llect
ion
.T
he im
po
rtan
t d
ecis
ion
mak
ers
or
thei
r vi
ews
mus
t b
e re
pre
sent
ed in
th
e w
ork
sho
p.
Inte
rvie
ws
Go
od
fo
r g
athe
ring
in-d
epth
info
rmat
ion
fr
om
res
po
nden
ts w
ho u
nder
stan
d a
n is
sue
wel
l. C
an in
clud
e b
oth
qua
ntita
tive
and
qua
litat
ive
dat
a.
May
no
t d
raw
out
new
idea
s o
r b
e fe
asib
le t
o
cond
uct
eno
ugh
for
a re
pre
sent
ativ
e sa
mp
le.
Can
be
cond
ucte
d f
ace-
to-f
ace
or
on
th
e p
hone
.
Cas
e st
udie
sC
an e
ffec
tivel
y ill
ustr
ate
an is
sue,
pro
mo
te a
pra
ctic
e, o
r ex
pla
in a
co
mp
lex
phe
nom
eno
n in
a s
pec
ific
cont
ext.
Cas
es m
ay n
ot
be
rep
rese
ntat
ive
even
if t
he
find
ing
s m
ay b
e us
eful
in a
ran
ge
of
cont
exts
.O
ften
use
d a
s p
art
of
a b
road
er
eval
uatio
n.
Med
ia a
naly
sis
May
pro
vid
e p
olic
y m
aker
s w
ith in
sig
ht in
to m
edia
co
vera
ge
of
po
litic
ally
sen
sitiv
e is
sues
.M
ay r
eflec
t m
edia
rat
her
than
key
sta
keho
lder
vi
ews
on
an is
sue.
New
s m
edia
is r
elat
ivel
y ea
sy a
nd
chea
p t
o o
bta
in.
Long
itud
inal
stu
die
sW
here
ind
ivid
uals
are
fo
llow
ed o
ver
time
to d
isco
ver
whe
ther
cha
nges
in, f
or
exam
ple
exp
osu
re t
o r
isk,
are
d
ue t
o a
sp
ecifi
c in
terv
entio
n.
Can
but
gen
eral
ly d
o n
ot
cons
ider
the
co
unte
r-fa
ctua
l, ie
– d
o n
ot
follo
w a
co
ntro
l g
roup
to
ass
ess
whe
ther
the
re a
re o
ther
ex
pla
natio
ns f
or
chan
ge.
Req
uire
bas
elin
e d
ata
to b
e co
llect
ed
bef
ore
pro
ject
is im
ple
men
ted
.
Qua
si-e
xper
imen
tal
des
igns
In
volv
e a
com
par
iso
n g
roup
. Are
use
d t
o m
easu
re
imp
act
whe
n ra
ndo
m a
lloca
tion
to t
reat
men
t o
r co
ntro
l g
roup
s is
no
t p
oss
ible
.
Sub
ject
s se
lf-se
lect
into
gro
ups,
whi
ch c
an
intr
od
uce
bia
s.R
equi
re p
roce
sses
fo
r ch
ecki
ng
inte
rven
tion
and
co
ntro
l gro
ups
are
equi
vale
nt p
rio
r to
an
inte
rven
tion
.
Two
use
ful r
efer
ence
s th
at d
etai
l po
ten
tial
met
ho
do
log
ies
and
wh
en t
o a
pp
ly t
hem
are
:
• R
efer
ence
Gu
ide
to E
valu
atin
g t
he
Eff
ecti
ven
ess
of
Str
ateg
ies
for
Pre
vent
ing
Wo
rk In
juri
es: H
ow
to
Sh
ow
Wh
eth
er a
Saf
ety
Inte
rven
tio
n R
eally
Wo
rks.
Ly
nd
a S
. Ro
bso
n, H
arry
S. S
han
no
n, L
ind
a M
. Go
lden
har
, An
dre
w R
. Hal
e. D
epar
tmen
t O
f H
ealt
h a
nd
Hu
man
Ser
vice
s, P
ub
lic H
ealt
h S
ervi
ce, C
ente
r fo
r D
isea
se C
on
tro
l an
d P
reve
nti
on
, Nat
ion
al In
stit
ute
fo
r O
ccu
pat
ion
al S
afet
y an
d H
ealt
h. A
pri
l 20
00
.
• A
RT
D R
epo
rt f
or
Wo
rk H
ealt
h a
nd
Saf
ety
Div
isio
n, W
ork
Co
ver
NS
W. C
ont
emp
ora
ry a
pp
roac
hes
an
d m
eth
od
s fo
r co
st e
ffec
tive
eva
luat
ion
. 20
15.
6
So
urc
es:
AR
TD
Re
po
rt f
or
Wo
rk H
eal
th a
nd
Saf
ety
Div
isio
n, W
ork
Co
ver
NS
W. C
on
tem
po
rary
ap
pro
ach
es a
nd
met
ho
ds
for
cost
eff
ect
ive
eval
uat
ion.
13
Fe
bru
ary
20
15. S
criv
en
, Mic
hae
l (19
91)
. Eva
luat
ion
Th
esau
rus.
4
th E
dit
ion
. Sag
e P
ub
licat
ion
s.
48 SAFEWORK NSW
6. Understanding, using and reporting evaluation findings
The main product of an evaluation is some form of report that is communicated to the intended audiences and used in line with the purpose. Likely audiences are NSW Government; SafeWork Executive, managers, operational staff; and external partners, for example regulatory agencies and industry partners.
6.1. Using evaluation results for service and product improvementOne of the fundamental reasons for evaluation is for improvement purposes. The results of evaluation studies should be used to improve services, products, harm prevention approaches and regulatory initiatives. As shown in the decision support tool, planning an evaluation also includes being explicit about how the evaluation will inform strategic and operational considerations about where directorates should direct efforts and resources in the future. Having appropriate governance structures in place will ensure that evaluation findings reach key decision makers and feed into other governance arrangements supporting the implementation of the Roadmap.
When planning how evaluation findings will be used, such a plan could include the following elements.7
• Plan to form an evaluation advisory group (see section 5.1), which includes both producers of evaluation findings and potential users of the findings, for example relevant operational staff and their managers, the evaluators and key external partners. The group should be used to discuss the implications of the interim findings for service improvement and produce recommendations to inform the development of annual business plans.
• Plan to present findings to the appropriate program governance committee to ensure the findings reach key decision makers and feed into other governance arrangements supporting the implementation of the Roadmap.
• Plan the timing of reporting of evaluation findings to coincide with business planning cycles and with the two-year review of the Roadmap.
• Identify how the report’s findings will be communicated to different stakeholders/audiences (see section 6.2).
6.2. Communication through evaluation reportsTo produce useful evaluation reports you should start planning the report early, including consulting communications professionals within SafeWork to seek advice on how to promote the findings to the different audiences.
The usefulness of a report depends upon its perceived credibility and how it is promoted. When promoting the report consideration should be given to the ‘What’s in it for me?’ principle from the perspective of the intended audience. This is particularly important for external partners who want reports that use plain English and that can be readily understood and distributed to their peers or members.8
Evaluation is a cycle, and it is important to complete the cycle of evaluation by reporting and using findings. This will keep intended audiences informed, assist managers to discuss findings with staff and consider the implications of the findings. Reports will also inform decision-making about the direction projects should take, for example about the:
• effectiveness of a risk mitigation or compliance strategy
• how to improve the effectiveness of delivery of your activities in order to improve results
• how to modify the project to more accurately target desired outcomes.
7 Grimshaw, Jeremy M., Eccles, Martin P., Lavis, John N., Hill, Sophie J., and Squires, Janet E. (2012). Knowledge translation of research findings. Implementation Science 127:50. and, Green, L.W., Ottoson, J.M., García, C., Hiatt, R.A., and Roditis, M.L.(2014). Diffusion theory and knowledge dissemination, utilization and integration. Frontiers in Public Health Services and Systems Research. 3(1).
8 Feedback from consultation interviews conducted by ARTD in February 2017.
WHS ROADMAP EVALUATION FRAMEWORK 49
6.3. Analysis of attribution and causal contributionOne of the things to consider in impact evaluations is how what you have implemented has caused things to change.
An outcomes evaluation is generally concerned not just with measured changes but ways in which a given program has contributed to those changes, and the influence of other factors. For the Roadmap, the achievement of shared outcomes is expected to be through the contribution of multiple interventions (such as those described in sector plans) across the organisation. Other factors could be the influence of social and economic changes and the direct actions of social partners.
The most rigorous methods for demonstrating program impact involve a counterfactual, but in the right circumstances – when it is feasible and ethical to have some form of valid comparison.
Contribution analysis (Mayne, 2012) explores causality in situations where experimental designs are not feasible, based on the program’s proposed theory of change/program logic and an analysis of this theory on the available evidence, including other influencing factors. There are six steps in this process.
1. Set out the cause-effect issue to be addressed.
2. Develop the postulated theory of change and risks to it, including rival explanations.
3. Gather the existing evidence on the theory of change.
4. Assemble and assess the contribution claim, and challenges to it.
5. Seek out additional evidence.
6. Revise and strengthen the contribution story.
Contribution analysis offers a practical approach to help assess whether a given intervention has contributed to an observed result.9 From a process of reasoned analysis, it is possible to infer whether a project is the most likely contributor to observed changes.
Causality is inferred from a four-step logic and evidence,10 outlined as follows:
• Step 1 – The intervention is based on a reasoned theory of change: the program logic and assumptions behind it are plausible, sound, informed by the literature and supported by key stakeholders.
• Step 2 – The activities of the intervention were implemented as outlined in the program logic.
• Step 3 – The theory of change is verified by evidence: the chain of expected results occurred and assumptions held.
• Step 4 – External factors (context and rival explanations) are assessed and shown not to have made a significant contribution or, if they did, their relative contribution is recognised.
Contribution analysis enables investigators to reach credible conclusions about the causal relationship between a project and its outcomes.
9 Better Evaluation, 2013, Approaches, http://betterevaluation.org/approaches
10 Mayne, J. 2012, Contribution analysis: Coming of Age? http://evi.sagepub.com/content/18/3/270
50 SAFEWORK NSW
6.4. Interpreting trends in lag indicators (long term policy outcomes)Any long term trends that are found in the lag indicators in the data (for example, reductions in injury rates, reductions in total losses paid or reductions in premiums charged based upon prior loss experience) must be interpreted with great sensitivity for the possibility of independent intervening variables that impact upon the outcome.
First, Australia specifically, and much of the rest of the developed world, is experiencing reductions in injury rates independent of the initiatives of this project.11 There has been much speculation about the cause of this reduction in reported accidents. Changes in the nature of work (such as more automation of dangerous work), the mix of work (such as a shift from manufacturing to less dangerous service provision as the mainstay of the economy), the nature of the workforce (such as more people working as contractors instead of employees, resulting in accidents often not being reported), and overall safety/liability consciousness on the part of tool and equipment manufacturers are some of the reason that are often cited.
Second, it is extraordinarily difficult to tease out the relative contributions to data trends created by one program in an environment where many programs are aimed at the reduction of injury. For instance, Safe Work Australia has made headway with public awareness of its national safety standard initiatives. Similarly, some impact has been felt from changes in director liability laws that require corporate boards to be more proactive and responsive to safety issues. Unions have also been active in the safety space, with at least one national union airing a series of television advertisements that feature the union's role in safety.
Third, individual compliance rates with any governmental initiative are influenced by a wide variety of factors that are less attributable to the content of the program than the manner in which the program is presented to the public.12 It is very difficult to separate the program content from the presentation of the content in ascertaining the impact of a program. That being said, the difference may not be one that is important from the perspective of achievement of the overall intention of the initiative, even if it impacts the assessment of the effectiveness of initiative components. For example, compliance activity may be the primary cause of behaviour change, or increased safety awareness brought about in part by compliance efforts may cause the observed change.
Finally, lag indicator trends must be viewed in the context of the larger environment. For instance, while injury rates in Australia are falling in many categories, the frequency of reported mental/psychological claims is up and the total expenditures for claims losses are also up.13 This presents a challenge for the delineation of the proper industry and harm categories for focus during the initiative.
Accurate and credible interpretation of trends in lag indicators can be advanced, although not entirely perfected, by analysis of all major known factors in the environment. The analysis, to be credible, must be transparent and as thorough as practicable, given resource and information limitations.
11 http://www.safeworkaustralia.gov.au/sites/SWA/Statistics/Documents/Table-1-1-number-frequency-rate-and-incidence-rate-of-serious-claims-by-occupation-2009-10-2013-14.pdf
12 Thaler, R. and Sunstein, C., Nudge: Improving Decisions About Health, Wealth, and Happiness. Penguin Group, N.Y., N.Y. 2009.
13 Comparison of Safe Work Australia reports for 2012–2013 (total cost $61.8 Billion) with the same report for 2008-2009 (total cost $34.1 Billion).
SW08641 0817