evaluating crime prediction maps - ucl.ac.uk€¦ · evaluating crime prediction maps. motivation...
TRANSCRIPT
![Page 1: Evaluating Crime Prediction Maps - ucl.ac.uk€¦ · Evaluating Crime Prediction Maps. Motivation ... Adepeju M., Rosser G. & Cheng T. (2016) Novel evaluation metrics for sparse spatiotemporal](https://reader030.vdocuments.net/reader030/viewer/2022041013/5ec40212b95f4c7bf27702c6/html5/thumbnails/1.jpg)
Big Data and Intelligent PolicingCPC Project Closing Workshop7th June 2016
MONSURU ADEPEJU & ROSSER GABRIEL
Evaluating Crime Prediction Maps
![Page 2: Evaluating Crime Prediction Maps - ucl.ac.uk€¦ · Evaluating Crime Prediction Maps. Motivation ... Adepeju M., Rosser G. & Cheng T. (2016) Novel evaluation metrics for sparse spatiotemporal](https://reader030.vdocuments.net/reader030/viewer/2022041013/5ec40212b95f4c7bf27702c6/html5/thumbnails/2.jpg)
Motivation
Given the increasing number of predictive hotspot methods, crime analysts often find it difficult to determine which method is most appropriate for different data scenarios.
For example,
• Which method is the best for predicting geographically constraint crime such as shoplifting?
• Which method is most robust to highly spatio-temporal sparse crime types?
![Page 3: Evaluating Crime Prediction Maps - ucl.ac.uk€¦ · Evaluating Crime Prediction Maps. Motivation ... Adepeju M., Rosser G. & Cheng T. (2016) Novel evaluation metrics for sparse spatiotemporal](https://reader030.vdocuments.net/reader030/viewer/2022041013/5ec40212b95f4c7bf27702c6/html5/thumbnails/3.jpg)
Proposed solution:
A systematic evaluation protocol by which the performance of multiple predictive methods can be assessed and compared
![Page 4: Evaluating Crime Prediction Maps - ucl.ac.uk€¦ · Evaluating Crime Prediction Maps. Motivation ... Adepeju M., Rosser G. & Cheng T. (2016) Novel evaluation metrics for sparse spatiotemporal](https://reader030.vdocuments.net/reader030/viewer/2022041013/5ec40212b95f4c7bf27702c6/html5/thumbnails/4.jpg)
Evaluation Framework
We combined measures of four different aspects of hotspot performance namely;
Accuracy,Compactness,Variability,Complementarity
![Page 5: Evaluating Crime Prediction Maps - ucl.ac.uk€¦ · Evaluating Crime Prediction Maps. Motivation ... Adepeju M., Rosser G. & Cheng T. (2016) Novel evaluation metrics for sparse spatiotemporal](https://reader030.vdocuments.net/reader030/viewer/2022041013/5ec40212b95f4c7bf27702c6/html5/thumbnails/5.jpg)
Evaluation metrics
1. Predictive accuracy (hit rate)
measures the number of crimes captured within the hotspot area
Hotspot area at time tn
Crime points at time tn+1
Pred. Acc. = 𝑛𝑜 𝑜𝑓 𝑐𝑟𝑖𝑚𝑒𝑠 𝑐𝑎𝑝𝑡𝑢𝑟𝑒𝑑
𝑇𝑜𝑡𝑎𝑙 𝑛𝑜 𝑜𝑓 𝑐𝑟𝑖𝑚𝑒
A method producing high predictive accuracy means more crimes can be intersected before they actually occur
![Page 6: Evaluating Crime Prediction Maps - ucl.ac.uk€¦ · Evaluating Crime Prediction Maps. Motivation ... Adepeju M., Rosser G. & Cheng T. (2016) Novel evaluation metrics for sparse spatiotemporal](https://reader030.vdocuments.net/reader030/viewer/2022041013/5ec40212b95f4c7bf27702c6/html5/thumbnails/6.jpg)
Evaluation metrics
2. Compactness Index (CI)
measures the ease at which a defined hotspot can be patrolled
From police standpoint, Map A is easier to patrol than Map B because of higher connectivity ratio of hotspot units
hotspotareas
hotspotareas
Map A Map B
![Page 7: Evaluating Crime Prediction Maps - ucl.ac.uk€¦ · Evaluating Crime Prediction Maps. Motivation ... Adepeju M., Rosser G. & Cheng T. (2016) Novel evaluation metrics for sparse spatiotemporal](https://reader030.vdocuments.net/reader030/viewer/2022041013/5ec40212b95f4c7bf27702c6/html5/thumbnails/7.jpg)
Evaluation metrics
3. Dynamic Variability Index (DVI)
measures the extent to which the predicted locations change between consecutive predictions
DVI helps to distinguish between different methods based on the type of crime risk they detect e.g. short-term risk, long-term risks.
𝑛 𝑛
repeated hotspot
disappearing hotspot
emerging hotspot
![Page 8: Evaluating Crime Prediction Maps - ucl.ac.uk€¦ · Evaluating Crime Prediction Maps. Motivation ... Adepeju M., Rosser G. & Cheng T. (2016) Novel evaluation metrics for sparse spatiotemporal](https://reader030.vdocuments.net/reader030/viewer/2022041013/5ec40212b95f4c7bf27702c6/html5/thumbnails/8.jpg)
Evaluation metrics
4. Complementarity
helps to reveal how much improvement a method made relative to other methods.
measures the extent to which different methods detect the same and/or different crimes
a b
c d
e f
g h
i
j
k l
m n
o
Venn diagram showing complementarity of four different methods; a – o indicate number of crimes
Adepeju M., Rosser G. & Cheng T. (2016) Novel evaluation metrics for sparse spatiotemporal point process – a crime case study STPP. Intl. J. Geog. Info. Sys. 1(22)
![Page 9: Evaluating Crime Prediction Maps - ucl.ac.uk€¦ · Evaluating Crime Prediction Maps. Motivation ... Adepeju M., Rosser G. & Cheng T. (2016) Novel evaluation metrics for sparse spatiotemporal](https://reader030.vdocuments.net/reader030/viewer/2022041013/5ec40212b95f4c7bf27702c6/html5/thumbnails/9.jpg)
Case Study - London Borough of Camden
Aim: To demonstrate the utility of the proposed evaluation framework
Predict-evaluate routine:
For each method, generate hotspots at day tn,
Evaluate the hotspot for one day ahead (tn+1)
Repeat for 100 consecutive days
Shoplifting
Dataset: 3 crime types of varied level of ST
sparseness
Burglary
Assault
![Page 10: Evaluating Crime Prediction Maps - ucl.ac.uk€¦ · Evaluating Crime Prediction Maps. Motivation ... Adepeju M., Rosser G. & Cheng T. (2016) Novel evaluation metrics for sparse spatiotemporal](https://reader030.vdocuments.net/reader030/viewer/2022041013/5ec40212b95f4c7bf27702c6/html5/thumbnails/10.jpg)
Results
mean hit rate, mean CI and mean DVI
Crime Type MethodAccuracy
Hotspot compactness
Variability
Mean Hit rate Mean CI Mean DVI
Shoplift
PSTSS 81.3 0.42 14.9
PKDE 74.3 0.55 2.7
SEPP 91.5 0.31 6.0PHotspot 85.1 0.37 19.2
Violence
PSTSS 46.5 0.46 10.8
PKDE 51.7 0.54 2.6
SEPP 59.7 0.12 4.5
PHotspot 52.2 0.32 21.1
Burglary
PSTSS 34.4 0.51 3.7
PKDE 38.8 0.50 2.3
SEPP 47.4 0.02 1.4
PHotspot 34.9 0.30 5.3
Evaluation metrics of Camden crime prediction at 20% coverage level.
![Page 11: Evaluating Crime Prediction Maps - ucl.ac.uk€¦ · Evaluating Crime Prediction Maps. Motivation ... Adepeju M., Rosser G. & Cheng T. (2016) Novel evaluation metrics for sparse spatiotemporal](https://reader030.vdocuments.net/reader030/viewer/2022041013/5ec40212b95f4c7bf27702c6/html5/thumbnails/11.jpg)
Shoplifting Violence Burglary
Undetected: 165Undetected: 155Undetected: 9
Shoplifting Violence Burglary
Undetected: 165Undetected: 155Undetected: 9
Results
Complementarity
Venn diagram showing the total number of crimes identified by each method at a fixed coverage of 20% in Camden
Shoplifting Violence Burglary
Undetected: 165Undetected: 155Undetected: 9
Total = 223 SEPP captures 213
Total = 526 PHotspot captures 220 SEPP captures 271
![Page 12: Evaluating Crime Prediction Maps - ucl.ac.uk€¦ · Evaluating Crime Prediction Maps. Motivation ... Adepeju M., Rosser G. & Cheng T. (2016) Novel evaluation metrics for sparse spatiotemporal](https://reader030.vdocuments.net/reader030/viewer/2022041013/5ec40212b95f4c7bf27702c6/html5/thumbnails/12.jpg)
Discussions
Trade-offs:
Predictive accuracy vs. Compactness (ease of patrol)
DVI reveals that certain methods are best suited for specific type of risk patterns. For example, PSTSS capture emerging risks patterns while PKDE captures persistent risk patterns
Complementarity suggests that results of some methods can be combined for better performance - ensemble predictions.