what works? targeting, testing & tracking (triple-t) amimb, glazier’s hall 29 october 2013...
TRANSCRIPT
![Page 1: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/1.jpg)
What Works?Targeting, Testing & Tracking
(Triple-T)
AMIMB, Glazier’s Hall29 October 2013
Lawrence W. ShermanInstitute of CriminologyCambridge University
![Page 2: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/2.jpg)
Forecasting Storms
• 1987 –Big False Negative• 2013—Big True Positive• Week in advance• Protective actions taken• Huge reduction in harm• Same in Eastern India 2013• Who was the hero?
![Page 3: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/3.jpg)
Who Was Vice-Admiral Robert Fitzroy?
![Page 4: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/4.jpg)
1819: Weather Was All Words
• Fitzroy changed the business model• ADDED numbers into it• Sharpened the words with scales, %• Improved accuracy over “experience”• Transformed qualitative forecasts• Into quantitatively-supported statements• Now we call them “algorithms”• See things coming much sooner
![Page 5: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/5.jpg)
Meteorological Office of the Board of Trade 1854
• Job: Publish Weather Data• Not Predict It• Fitzroy invented term “forecasting”• Controversial idea• Not 100% accurate• Royal Society Attacked It• Budget Cut; forecasts stopped• Fitzroy committed suicide
![Page 6: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/6.jpg)
Prevention Finally Won• Falling Barometer Gale Warnings• Fishing ships told to stay in port• Owners objected; Practice discontinued• Deaths from gales; Prevention revived
![Page 7: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/7.jpg)
What’s YOUR Job?
• Measuring Harm?• Preventing Harm? • Reducing Harm?• Correcting Harm?• Redressing Wrongs?
What is your business model?
Nor for making money.
Just for doing the job.
![Page 8: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/8.jpg)
Words--not many Numbers
• Every prison is different• Every human life is important• All rights must be protected
BUT....• Which prisons are worst?• Which are about to explode in violence?• Which will have the most suicides?• How can algorithms help prevent harm?
![Page 9: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/9.jpg)
AMIMB Website Einstein Quote
• Using same method• Expect different result?• Is there a new method IMBs could use?• Are there other business models?• Could they get better results? • Could they ADD numbers to add VALUE?
![Page 10: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/10.jpg)
Evidence-Based Practice
1. An Idea:
“...Practices should be based on
scientific evidence about what works best.”
(Sherman, 1998)
2. An Analytic Framework
“A standard for all....strategies: using scientific evidence to target, test and track ...[all] practices” (Sherman, 2013)
![Page 11: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/11.jpg)
Words AND Numbers
• AMIMB Practical Guide Has Many Words• Many numbers requested• Not clear how they are to be gathered• Whether comparisons could be reliable• What is the logic model by which• EACH IMB REPORT CAN REDUCE
HARM?
![Page 12: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/12.jpg)
Same Problem for HM Inspectors
• Police since 1856• Prisons? Probation?• “Efficiency and Effectiveness”• Case by case• Word by word• HMI Police developed a “report card”• But measures were not scientific• Not “risk-adjusted”
![Page 13: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/13.jpg)
Daniel Kahneman
![Page 14: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/14.jpg)
QUICK! What emotion does this person feel?
![Page 15: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/15.jpg)
QUICK! How Much is This Product?
17 X 24 = ???
![Page 16: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/16.jpg)
Thinking, Fast and….Slower
• Reading the face—fast!
• Multiplying two numbers—slow (408)
• Why is that important to monitoring?
• It adds theory about how we think, decide, act
![Page 17: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/17.jpg)
Two Cognitive Systems:Overlapping
System I FAST
Intuitive
Automatic
Effortless
Associative
Rapid
Opaque Process
Skilled
System IISLOW
Reflective
Controlled
Effortful
Deductive
Slow
Self-Aware
Rule-Following
![Page 18: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/18.jpg)
Good News, Bad News
Good News:• Most decisions are made with System I• System I conserves energy, time• Most System I decisions are right--driving
Bad News:• Many important decisions are “wrong”• Many could be “right” if we used System II• We resist System II because it is “costly,” tiring
![Page 19: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/19.jpg)
Slow Thinking About Strategy
TargetingAiming for biggest impact
TrackingMeasuring BOTH policing and crime
TestingDeciding what works
![Page 20: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/20.jpg)
Targeting---PREDICT
Focus
•Issues, situations, processes
•Units, managers, problem leaders
Classify
•Concentrations
•Causes
Prioritize
•Greatest impact
•Best chance of success
![Page 21: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/21.jpg)
Clinical vs. Statistical: Kahneman Chapter 21Began Career with
Paul Meehl (1954)200 Replications
Since ThenFormulas always
beat (60%) or equal (40%) human judgment—latter is far more costly
Crime, parole, air pilot errors, credit risk
Wine value forecasts Cot Death risk
Paul Meehl
![Page 22: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/22.jpg)
3 ways of PredictingQualitative1. Clinical (System 1~2?; not transparent)
Statistical:
2. checklist with validation
3. supercomputer data mining
![Page 23: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/23.jpg)
High Risk (2%)
Neither High nor Low Risk (38%)
Low Risk (60%)
![Page 24: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/24.jpg)
High Risk 2% vs. Bottom 60%Two Years From Forecast Date
Charges for Any Offence 8 X more
Charges Serious Offence 10 X more
Charges Murder or Attempt 75 X more
![Page 25: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/25.jpg)
TestingComparing two methodsSame kind of problems
THEN,.... ASKING:
Which one works better?Which one costs less?Which one gets best result for
same cost?
![Page 26: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/26.jpg)
Testing DefinedA Fair Comparison Between two different methodsE = Experimental Method (new)C= Control Method (current) All else equalWhich one is better?By what criteria?
![Page 27: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/27.jpg)
What Kind of Evidence?Not This Kind But This Kind
![Page 28: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/28.jpg)
But what is good evidence?What constitutes a good test?
Many bad tests
More intuitive than quantitative
“Illusion of Validity”
![Page 29: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/29.jpg)
Restorative Justice Experiment:Did Program Cause Crime Drop?
71% 71%
49%
100%
20%
40%
60%
80%
100%
120%
140%
Yr (-2) Yr (-1) Yr (+1) Yr (+2)
Line 1
DCOffenders (n =62)
![Page 30: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/30.jpg)
Inferring Cause From Trend?Post Hoc Ergo
Propter Hoc?
After this, then because of this?
![Page 31: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/31.jpg)
Or would it have dropped anyway?“Natural” Trend“History”Other factors“Spurious” explanationsThat can be eliminated—give confidence
All ruled out by randomized controlled trials
(RCTs)
![Page 32: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/32.jpg)
Randomized Controlled Trial RCT:COMPARISON or NET difference
101%
121%
71% 71%
28%
49%
100%
20%
40%
60%
80%
100%
120%
140%
Yr (-2) Yr (-1) Yr (+1) Yr (+2)
CourtOffenders(n=59)
DCOffenders (n =62)
![Page 33: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/33.jpg)
CONTROL Group A sample that measures what would
happen at the same time and place without the intervention being introduced to an otherwise identical
EXPERIMENTAL Group
![Page 34: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/34.jpg)
AFP/ACT ANU Experiment:Did Program Cause Crime Drop?
71% 71%
49%
100%
20%
40%
60%
80%
100%
120%
140%
Yr (-2) Yr (-1) Yr (+1) Yr (+2)
Line 1
DCOffenders (n =62)
![Page 35: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/35.jpg)
Inferring Cause From Trend?
• Post Hoc Ergo Propter Hoc?
• After this, then because of this?
![Page 36: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/36.jpg)
Or would it have dropped anyway?
• “Natural” Trend• “History”• Other factors• “Spurious” explanations• That can be eliminated—give confidence
All ruled out by randomized controlled trials
(RCTs)
![Page 37: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/37.jpg)
Randomized Controlled Trial RCT:COMPARISON or NET difference
101%
121%
71% 71%
28%
49%
100%
20%
40%
60%
80%
100%
120%
140%
Yr (-2) Yr (-1) Yr (+1) Yr (+2)
CourtOffenders (n=59)
DCOffenders (n =62)
![Page 38: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/38.jpg)
CONTROL Group
• A sample that measures what would happen at the same time and place without the intervention being introduced to an otherwise identical
EXPERIMENTAL Group
![Page 39: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/39.jpg)
Tracking
Crime Where When
Policing Patrol, POP Arrests
Match? Ratios Trends
![Page 40: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/40.jpg)
Tracking in Politics
• Obama’s “Cave”• Tracking volunteer activity—500 offices• Contacting voters• Determining who supports Obama• Determining when & how they will vote• Arranging transport to the polls • Scheduling election day drivers • “Ground Game” cost $100 Million
![Page 41: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/41.jpg)
3. Tracking in Prisons
• Liebling’s Quality of Prison Life Measures
• What are the trends in
--inputs (resources)
--outputs (activities)
--outcomes (results)
![Page 42: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/42.jpg)
AMIMB-NOMS COMPSTAT?
![Page 43: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/43.jpg)
Tracking With Evidence
• Discuss• Criticize• Problem-Solve• Use and misuse of data• Refinement through trial and error • Technology making change easier
![Page 44: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/44.jpg)
Sherman: 1998
![Page 45: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/45.jpg)
![Page 46: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/46.jpg)
The Rise of Evidence?
• More Evidence is Available• More Evidence is Demanded• Budget cutbacks make evidence relevant• But there are only early adopters• A tipping point may some soon• The pace is quickening
![Page 47: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/47.jpg)
How Can IMBs Be Evidence-Based?
1. Quantify current reports
2. Design new—fewer—measures
3. Test different models of IMB work
![Page 48: What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University](https://reader035.vdocuments.net/reader035/viewer/2022062318/551b35c7550346dd1a8b50b7/html5/thumbnails/48.jpg)
What Works?Targeting, Testing & Tracking
(Triple-T)
THANK YOU
Lawrence W. ShermanInstitute of CriminologyCambridge University