the application of “value engineering” tools to risk assess the outputs of an nsi
DESCRIPTION
The application of “Value Engineering” tools to risk assess the outputs of an NSI. Graham Sharp Manager, Continuous Improvement Zone ONS, UK [email protected] 0044 1633 456742. Problem & goal statements. Problem Statement No consistent means of assessing the risks of our - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: The application of “Value Engineering” tools to risk assess the outputs of an NSI](https://reader035.vdocuments.net/reader035/viewer/2022070419/56815b5a550346895dc9400f/html5/thumbnails/1.jpg)
The application of “Value Engineering” tools to risk assess the outputs of an NSI
Graham Sharp
Manager, Continuous Improvement Zone
ONS, UK
0044 1633 456742
![Page 2: The application of “Value Engineering” tools to risk assess the outputs of an NSI](https://reader035.vdocuments.net/reader035/viewer/2022070419/56815b5a550346895dc9400f/html5/thumbnails/2.jpg)
Problem & goal statements
Problem Statement• No consistent means of assessing the risks of our statistical outputs in a standardised way • Need a strategic approach to be taken to prioritise improvements
Goal Statement
To develop a risk assessment methodology and deliver a scored risk assessment of ONS statistical outputs by the end of 2012.
![Page 3: The application of “Value Engineering” tools to risk assess the outputs of an NSI](https://reader035.vdocuments.net/reader035/viewer/2022070419/56815b5a550346895dc9400f/html5/thumbnails/3.jpg)
Scope & solution requirements
In Scope:
• Evaluate all statistical outputs and the statistical system(s)/tool(s) which are used to produce them• Consider the entire GSBPM• Consider a number of dimensions of risk
Solution requirements:
• Measure the risk of each ONS output • Outputs to be compared against each other• Allow drill down capabilities to identify root causes of scores• Simple to apply • Capable of self-assessment by output managers
![Page 4: The application of “Value Engineering” tools to risk assess the outputs of an NSI](https://reader035.vdocuments.net/reader035/viewer/2022070419/56815b5a550346895dc9400f/html5/thumbnails/4.jpg)
High level project stages
• Agree coverage/scope of project with Directors• Communication with staff• Design solution dimensions and weighting • Set up solution template• Pilot with business areas• Refine tool if necessary• Collect information from all business areas• Collate into template tool and produce RAG status
per system/output• QA results• Produce analysis output
![Page 5: The application of “Value Engineering” tools to risk assess the outputs of an NSI](https://reader035.vdocuments.net/reader035/viewer/2022070419/56815b5a550346895dc9400f/html5/thumbnails/5.jpg)
Value Engineering
Wikipedia Definition:
“Value engineering (VE) is a systematic method to improve the "value" of goods or products and services by using an examination of function”
Applied to statistical outputs, it provides a systematic risk assessment against a number of dimensions
![Page 6: The application of “Value Engineering” tools to risk assess the outputs of an NSI](https://reader035.vdocuments.net/reader035/viewer/2022070419/56815b5a550346895dc9400f/html5/thumbnails/6.jpg)
Dimensions – Are they ‘fit for purpose’?
Sources
Methods
Systems
Processes
Quality
Users & Reputation
People
Census dataAdmin dataSurvey data
System named and reason for red/amber provided
European dimensions of Relevance; Accuracy; Timeliness & Punctuality; Accessibility & Clarity; Comparability; Coherence
Are there sufficient skilled and trained people working on the output?
Data acquisition /Questionnaire design; Coverage of data; Processing, edit & imputation; Analysis; Disclosure
Data collection & preparation; Results & analysis
User feedback; Future user needs; Reputation
![Page 7: The application of “Value Engineering” tools to risk assess the outputs of an NSI](https://reader035.vdocuments.net/reader035/viewer/2022070419/56815b5a550346895dc9400f/html5/thumbnails/7.jpg)
Process for first implementation
• Review of template with each Deputy Director (DD)• 3 pilot sessions with outputs managers• Updated template• Self assessment by output managers• Quality assured by DD, data collection areas and
Methodologists• Importance weights reviewed by Directors• Challenges responded to by output managers/DDs• Results collated
![Page 8: The application of “Value Engineering” tools to risk assess the outputs of an NSI](https://reader035.vdocuments.net/reader035/viewer/2022070419/56815b5a550346895dc9400f/html5/thumbnails/8.jpg)
Scoring process
No issues or N/A 0 N/A Some improvements possible 3 Comments In need of attention 9 Comments
Output Systems Summary score
A Sub 1 Sub 2 Sub 3 9
B Sub 1 Sub 2 Sub 3 3
C Sub 1 Sub 2 Sub 3 3
Output
Sources
Methods
Systems
Processes
Quality
Users & Reputation
People
Summary score
Weighting
Composite score
A 0 3 9 3 3 0 0 18 3 54
B 3 0 3 3 9 3 3 24 2 48
C 0 3 3 3 0 0 3 12 1 12
DD to complete:Confirm the relative importance of the output to users and the impact to ONS reputation if results were erroneous.Low =1Med = 2High =3
![Page 9: The application of “Value Engineering” tools to risk assess the outputs of an NSI](https://reader035.vdocuments.net/reader035/viewer/2022070419/56815b5a550346895dc9400f/html5/thumbnails/9.jpg)
Overall assessment
2012 2013% red overall
21.4% 18.7%% amber overall
46.8% 48.1%% green overall
32.0% 33.2%
Baseline measure of ‘% red overall’ used as a KPI in ONS business planning
Data collection carried out in November 2012 and again in November 2013
![Page 10: The application of “Value Engineering” tools to risk assess the outputs of an NSI](https://reader035.vdocuments.net/reader035/viewer/2022070419/56815b5a550346895dc9400f/html5/thumbnails/10.jpg)
Highest scoring outputs
Analysis of scores in November 2012 and November 2013 allowed identification of top ten scoring outputs – i.e. highest risk
Reasons for movements over the year also analysed to identify existing mitigating actions and identification of what remains to be addressed
![Page 11: The application of “Value Engineering” tools to risk assess the outputs of an NSI](https://reader035.vdocuments.net/reader035/viewer/2022070419/56815b5a550346895dc9400f/html5/thumbnails/11.jpg)
Analyse - Boxplots
Comparison of scores after 1 year• Identify improvements made• Confirm new candidate surveys for improvement• Review cause of outliers and corrective action required
Boxplots of weighted scores – C&P
BIBOP BOD C&P Ops LMD NACD Prices PSH SSD0
20
40
60
80
100
120
140
160
180
200 2012 2013
BIBOP BOD C&P Ops LMD NACD Prices PSH SSD0
20
40
60
80
100
120
140
160
180
Weig
hte
d s
co
re
Median 25th 75th Mean Extreme Outliers
![Page 12: The application of “Value Engineering” tools to risk assess the outputs of an NSI](https://reader035.vdocuments.net/reader035/viewer/2022070419/56815b5a550346895dc9400f/html5/thumbnails/12.jpg)
Findings by Dimensions - Processes
Red16%
Amber47%
Green37%
BIBOP
LMD
BODM
NW
Popula
tion
Pubse
c & H
ouse
holds
Crime,
regio
nal &
dat
a ac
cess
Health
& lif
e ev
ents
Social
sur
veys
0
2
4
6
No. of red outputs by Division
Pubsec & householdsBIBOP
LMDMNW
Crime, regional & data accessBOD
PopulationSocial surveys
Health & life eventsNACD
Public Policy AnalysisOCEA
C&P Ops Prices
0% 10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Percentage of outputs scoring RAG by category
Number of red outputs by Division for this category
Percentage of RAG outputs for each Division for this category
![Page 13: The application of “Value Engineering” tools to risk assess the outputs of an NSI](https://reader035.vdocuments.net/reader035/viewer/2022070419/56815b5a550346895dc9400f/html5/thumbnails/13.jpg)
Use of analysis to date (1)
• Prioritisation of National Statistics Quality Reviews
• Input to survey action plans - identifying and prioritising key improvements required
• Identifying local continuous improvement initiatives
• Prioritising developments and influencing budget allocations
• Sense checking where we are currently investing in developments
![Page 14: The application of “Value Engineering” tools to risk assess the outputs of an NSI](https://reader035.vdocuments.net/reader035/viewer/2022070419/56815b5a550346895dc9400f/html5/thumbnails/14.jpg)
Use of analysis to date (2)
• Deploying our skilled people to reduce risks in key areas
• Improving communications on outputs
• Highlighting where we need careful stakeholder handling
![Page 15: The application of “Value Engineering” tools to risk assess the outputs of an NSI](https://reader035.vdocuments.net/reader035/viewer/2022070419/56815b5a550346895dc9400f/html5/thumbnails/15.jpg)
Conclusion
• Model meets the intended purpose
• Gaining in popularity and application
• Has become a key tool in the risk assessment of ONS outputs
• Need to be aware that this is based on self assessment, but mitigating actions in place
• Should be used as part of a wider range of risk & quality assessment tools
![Page 16: The application of “Value Engineering” tools to risk assess the outputs of an NSI](https://reader035.vdocuments.net/reader035/viewer/2022070419/56815b5a550346895dc9400f/html5/thumbnails/16.jpg)
Questions