a framework for statistical performance
DESCRIPTION
Presentation at the Royal Statistical Society's International Conference, Newcastle, September 2013. Paul Askew speakingdata.org.uk [email protected] A Framework Understating Statistical Performance This paper presents a framework for understanding, managing and presenting statistical performance data. It provides a manageable, but multidimensional way of organising a range of possible analysis and assessment to provide a rounded and balanced picture for effective understanding and communication. This is founded on a broad scope of assessment, based around three key analytical elements: trend, benchmark and target. That is the first multidimensional perspective. Each of those three key elements are then further disaggregated in a range of appropriate ways to provide a second, more disaggregate, multidimensional analytical assessment. There is then a third more tailored and specific multidimensional disaggregation to provide additional detailed statistical insight. This draws on practical application of this framework in a range of sectors including health, policing and education, and across the public sector spectrum including Central Government, regulation and local service delivery, and in both strategic and operational environments. This provides an overall framework to manage the communication and understanding of statistical performance, and focussed on public data. It provides a framework to build a balanced communication of statistical analysis and messages, and at the same time it provides the user and recipient of statistics and statistical analysis with a framework to both understand and question the scope and content of statistical communication.TRANSCRIPT
A Framework for Understanding Statistical Performance
Paul Askew
CONFERENCE2-5 SEPTEMBER 2013
NEWCASTLE
1. Introduction
2. Framework – the “Why”
Operational Drivers
Current Strategic Drivers
3. Framework – the “How”
Macro level
Analytical level
Outline
1. Scope….A framework for
Managing Statistics about performance
(rather than performance of statistical techniques)
2. Operational Origins
• More about practical drivers and process
• Utility….target setting, performance improvement
3. Distilling application and development across sectors….
• Criminal justice, regulation, education, health
• It really matters….safety, housing, education….
1. Introduction
1. Introduction
Operational Delivery
Methodological Leadership
1. Scope….A framework for
Managing Statistics about performance
(rather than performance of statistical techniques)
2. Operational Origins
• More about practical drivers and process
• Utility….target setting, performance improvement
3. Distilling application and development across sectors….
• Criminal justice, regulation, education, health
• It really matters….safety, housing, education….
1. Introduction
1. Introduction
2. Framework – the “Why”
Operational Drivers
Current Strategic Drivers
3. Framework – the “How”
Macro level
Analytical level
Outline
Data
Eviden
ce
Decisi
ons
1. It actually matters to people – safety, home, education
2. Performance Regime – broad scope, high profile, deep drill down
3. “Multi-multi” dimensional – both of measures and assessments
4. Statistics meaning – datum, summary, technique
5. Targets - legal, audited, collaborative!
6. Performance Pantomime
7. Less about techniques, more about process
8. Operational Delivery – police, health, regulation…
2. Why - Operational Drivers
“Burglary is down compared to last month”
“Yes but it’s up compared the same month last year”
“Yes but it’s down overall for the financial year to date”
“Yes but its’ up for the calendar year so far”
“Yes but we’re still less better than our neighbours”
“Yes but they are reducing faster than we are this year”
“Yes but we’re still under (over) target”.
etc………….
1. It actually matters to people – safety, home, education
2. Performance Regime – broad scope, high profile, deep drill down
3. “Multi-multi” dimensional – both of measures and assessments
4. Statistics meaning – datum, summary, technique,
5. Targets - legal, audited, collaborative!
6. Performance Pantomime
7. Less about techniques, more about process
8. Operational Delivery – police, health, regulation…
2. Why - Operational Drivers
Smoothed Data or Real Data
Highs and lowsHigh and low
DecreasingIncreasing Increasing convergence
Decreasing convergence
Three month step Six month stepTwo month step
Smoothed Data – 12 month rolling average
Smoothed Data
Example Real Data
Notes: Real data for 12 months, previous 12 months is exactly the same, to create 12 month rolling average (mean).
This smoothed data is derived from any of these underlying
raw data examples.
1. Data, Evidence, Decisions… Impact, Value.
2. Big & Open & Now data
3. Tactical vs. Strategic focus
4. Key Strategies…Communication emphasis - ONS, RSS…
5. Underlying Numeracy and statistical literacy
6. Policy Perception Gap
7. Data Science – Shakespeare review, Open Data, UKSA…
8. Austerity World - Effective (right thing) & Efficient (right way)
2. Why - Current and Strategic Drivers
Scope - Detail - Volume
Data.gov…10K
1. Data, Evidence, Decisions… Impact, Value.
2. Big & Open & Now data
3. Tactical vs. Strategic focus
4. Key Strategies…Communication emphasis - ONS, RSS…
5. Underlying numeracy and statistical literacy
6. Policy Perception Gap
7. Data Science – Shakespeare review, Open Data, UKSA…
8. Austerity World - Effective (right thing) & Efficient (right way)
2. Why - Current and Strategic Drivers
Words
Numbers
Data
Eviden
ce
Decisi
ons
1. Data, Evidence, Decisions… Impact, Value.
2. Big & Open & Now data
3. Tactical vs. Strategic focus
4. Key Strategies…Communication emphasis - ONS, RSS…
5. Underlying Numeracy and statistical literacy
6. Policy Perception Gap
7. Data Science – Shakespeare review, Open Data, UKSA…
8. Austerity World - Effective (right thing) & Efficient (right way)
2. Why - Current and Strategic Drivers
Skills for Life Survey 2011 (England)Department for Business Innovation and Skills
• Literacy Improving while Numeracy declining
Numeracy • 26% to 22% (7.5m
adults) with GCSE+
• 17m adults at primary school level
The numeracy challenge is big and getting bigger…%
Adu
lts a
t GC
SE
+ L
evel
s
A Framework for Understanding Statistical Performance
Paul Askew
1. Data, Evidence, Decisions… Impact, Value.
2. Big & Open & Now data
3. Tactical vs. Strategic focus
4. Key Strategies…Communication emphasis - ONS, RSS…
5. Underlying Numeracy and statistical literacy
6. Policy Perception Gap
7. Data Science – Shakespeare review, Open Data, UKSA…
8. Austerity World - Effective (right thing) & Efficient (right way)
2. Why - Current and Strategic Drivers
1. Introduction
2. Framework – the “Why”
Operational Drivers
Current Strategic Drivers
3. Framework – the “How”
Macro level
Analytical level
Outline
3. How - Macro
DATA - inputs -
ANALYSIS- process -
PRODUCTS- outputs -
INSIGHT- outcomes -
1. Purpose
2. Require-
ments
4. Design
3. Constr-
aints
5. Defiine
6. Specify
8. Record
7. Collect
9. Entering
10.Process
12. Storage
11. Validate
2.Tools
1. Data
3.Skills
4.Capacity
5.Question
6.Inclinat-
ion
Plan
Implement
Manage
3.Target
2. Bench-mark
1. Trend
ANALYSISComp-
aritorsTime
Periods
DATA
Time Periods
Keys Message
INSI
GH
T
PRODUCTS
Cover the
angles
Lift Pitch EvidenceSummary
Comms
Stake-holders
Synthesis
- inputs -
- process --
- outputs -
- out
com
es - Analysis Strategy
Words
Graphics
Numbers
1. Purpose
2. Require-
ments
4. Design
3. Constr-
aints
5. Defiine
6. Specify
8. Record
7. Collect
9. Entering
10.Process
12. Storage
11. Validate
2.Tools
1. Data
3.Skills
4.Capacity
5.Question
6.Inclinat-
ion
Plan
Implement
Manage
3.Target
2. Bench-mark
1. Trend
ANALYSISComp-
aritorsTime
Periods
DATA
Time Periods
Keys Message
INSI
GH
T
PRODUCTS
Cover the
angles
Lift Pitch EvidenceSummary
Comms
Stake-holders
Synthesis
- inputs -
- process --
- outputs -
- out
com
es - Analysis Strategy
Words
Graphics
Numbers
OPE
NOPEN
OPEN
OPEN
II = (d)D x (t)T x s(S) x (c)C x (q)Q x (i)IThe factors:
D Data: Right data? Enough of it? Good enough?T Tools: Have any? Right ones?S Skills: Have any? Right ones?C Capacity: How much? Realistic?Q Question: Specific question to answer or issues to address I Inclination: Desire and drive to want to address the issues
d,t,s,c,q,i Relative weights
Askew Analytical Insight Index
1. Purpose
2. Require-
ments
4. Design
3. Constr-
aints
5. Defiine
6. Specify
8. Record
7. Collect
9. Entering
10.Process
12. Storage
11. Validate
2.Tools
1. Data
3.Skills
4.Capacity
5.Question
6.Inclinat-
ion
Plan
Implement
Manage
3.Target
2. Bench-mark
1. Trend
ANALYSISComp-
aritorsTime
Periods
DATA
Time Periods
Keys Message
INSI
GH
T
PRODUCTS
Cover the
angles
Lift Pitch EvidenceSummary
Comms
Stake-holders
Synthesis
- inputs -
- process --
- outputs -
- out
com
es - Analysis Strategy
Words
Graphics
Numbers
OPE
NOPEN
OPEN
OPEN
3.Target
2. Bench-mark
1. Trend
Comp-aritors
Time Periods
Time Periods
3. How – Analytical Level
0. Snapshot
0. Snapshot - we have a number which is important to us
1. Trend - what’s happening to our measure over time
2. Benchmark – how this compares to others
2a. Trend for the comparison to others
3. Target – the trajectory for our measure
3a. – the comparison trajectory 3.2. 1. 0.
3. How – Analytical Level
0. Snapshot - we have a number which is important to us
1. Trend - what’s happening to our measure over time
2. Benchmark – how this compares to others
2a. Trend for the comparison to others
3. Target – the trajectory for our measure
3a. – the comparison trajectory 3.2. 1. 0.
3. How – Analytical Level
0. Snapshot - we have a number which is important to us
1. Trend - what’s happening to our measure over time
2. Benchmark – how this compares to others
2a. Trend for the comparison to others
3. Target – the trajectory for our measure
3a. – the comparison trajectory 3.2. 1. 0.
3. How – Analytical Level
0. Snapshot - we have a number which is important to us
1. Trend - what’s happening to our measure over time
2. Benchmark – how this compares to others
2a. Trend for the comparison to others
3. Target – the trajectory for our measure
3a. – the comparison trajectory 3.2. 1. 0.
3. How – Analytical Level
0. Snapshot - we have a number which is important to us
1. Trend - what’s happening to our measure over time
2. Benchmark – how this compares to others
2a. Trend for the comparison to others
3. Target – the trajectory for our measure
3a. – the comparison trajectory 3.2. 1. 0.
3. How – Analytical Level
0. Snapshot - we have a number which is important to us
1. Trend - what’s happening to our measure over time
2. Benchmark – how this compares to others
2a. Trend for the comparison to others
3. Target – the trajectory for our measure
3a. – the comparison trajectory 3.2. 1. 0.
3. How – Analytical Level
t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+40
20
40
60
80
100
120
140
160
Value
Time
0. Snapshot – we have a number which is important to us
t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+40
20
40
60
80
100
120
140
160
Value
Time
1. Trend – what’s happening over time
t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+40
20
40
60
80
100
120
140
160
Value
Time
2. Benchmark – how this measures compares to others
t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+40
20
40
60
80
100
120
140
160
Value
Time
2a. Trend for the comparison to others
t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+40
20
40
60
80
100
120
140
160
Value
Time
3. Target - the trajectory for our measure
t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+40
20
40
60
80
100
120
140
160
Value
Time
3a. Target - Trajectory for the comparison to others
1. Introduction
2. Framework – the “Why”
Operational Drivers
Current Strategic Drivers
3. Framework – the “How”
Macro
Analytical
Outline
A Framework for Understanding Statistical Performance
Paul Askew
CONFERENCE2-5 SEPTEMBER 2013
NEWCASTLE
Thank You