to crawl before we run: optimising therapies with aggregated data chris evans, michael barkham, john...
TRANSCRIPT
To crawl before we run:optimising therapies with aggregated data
Chris Evans, Michael Barkham,
John Mellor-Clark, Frank Margison, Janice Connell
AimsPanel aim is to help bridge the gap between
researchers and practitionersSpecifically, to promote new forms of “practice
based evidence” (PBE) which work in and across that gap and which complement EBP
This paper aims to present low sophistication, service oriented methods to complement the HLM and other sophisticated methods that Wolfgang, Zoran and many others have developed
Specific aims for this presentation
Show realities of routine data collectionShow the magnitude of service level variationArgue that simple service level analyses can
help us learn from treatment failuresComputer processing is needed by most
services/practitioners but is alien to many, two methods of computer processing available for CORE
For now, confidence intervals and graphical data presentations may be the “zone of proximal development”
The dataset
6610 records (from >12k):33 primary care NHS services40 to 932 records per serviceAnonymised, voluntaryFour components to the data:
Therapist completed CORE-ATherapy Assessment Form (TAF)End of Therapy Form (EOT)
Client completed CORE-OMAt assessment and end of therapy or follow-up
CORE-A TAF
CORE-A EOT
CORE-OM
CORE-PC version of CORE-OM
“It’s really simple and easy to use. I’m not very computer literate, but I’d got to grips with it in less than an hour”
Plotting data: simple proportion
Services
%
02
04
06
08
01
00
% with the second CORE-A (EOT) form1
49
6
2
18
23
13
85
30
08 13
89
13
62
10
40
7 44
8 10
87
15
64
1 38
7 18
05
4
5
92
7 81
2
10
86
13
79
91
5 32
9
13
08
96
8
18
19
12
32
16
57
95
0 17
44
20
21
53
2 20
3
Plotting data: reference lines
Services
%
02
04
06
08
01
00
% with the second CORE-A (EOT) form1
49
6
2
18
23
13
85
30
08 13
89
13
62
10
40
7 44
8 10
87
15
64
1 38
7 18
05
4
5
92
7 81
2
10
86
13
79
91
5 32
9
13
08
96
8
18
19
12
32
16
57
95
0 17
44
20
21
53
2 20
3
Plotting data: add CI for sites
Services
%
02
04
06
08
01
00
% with the second CORE-A (EOT) form1
49
6
315
2
932
18
23
40
13
85
69
30
08
135
13
89
128
13
62
49
10
40
153
7
430
44
8
55
10
87
51
15
64
60
1286
38
769
18
05
113
4
168
5
164
92
7
75
81
2
102
10
86
62
13
79
165
91
5
102
32
9
323
13
08
142
96
8
639
18
19
141
12
32
690
16
57
98
95
0
203
17
44
101
20
21
77
53
2
300
20
3
173
Plotting data: add summary
Services
%
02
04
06
08
01
00
% with the second CORE-A (EOT) form1
49
6
315
2
932
18
23
40
13
85
69
30
08
135
13
89
128
13
62
49
10
40
153
7
430
44
8
55
10
87
51
15
64
60
1
2863
87
691
80
5113
4
168
5
164
92
7
75
81
2
102
10
86
62
13
79
165
91
5
102
32
9
323
13
08
142
96
8
639
18
19
141
12
32
690
16
57
98
95
0
203
17
44
101
20
21
77
53
2
300
20
3
173
Overall proportion = 72.1%Maximum = 98.8%Minimum = 33%Ratio, max:min = 2.99Chi square = 1181.75 d.f. = 32 p = 0
Number "significantly" high 13Number "significantly" low 10Number "significantly" different 23
Getting data (2): CORE-OM 1
Services
%
02
04
06
08
01
00
% with CORE-OM at assessment1
08
7
51
5
164
30
08
135
1
286
4
168
13
08
142
7
430
18
23
40
53
2300
13
89
128
96
8
639
10
40
153
18
05
113
32
9
323
20
21
77
13
85
69
2
932
95
0
203
12
32
690
15
64
60
18
19
141
14
96
315
81
2
102
38
7
69
92
7
75
16
57
98
20
3
173
10
86
62
91
5
102
13
79
165
13
62
49
44
8
55
17
44
101
Overall proportion = 88.3%Maximum = 100%Minimum = 39.2%Ratio, max:min = 2.55Chi square = 396.43 d.f. = 32 p = 0
Number "significantly" high 13Number "significantly" low 9Number "significantly" different 22
Getting data (3): CORE-OM 2
Services
%
02
04
06
08
0
% with final CORE-OM5
32
300
14
96
315
13
85
69
44
8
55
10
87
51
18
23
40
10
40
153
30
08
135
13
89
128
20
21
77
13
62
49
7
430
5
164
38
7
69
2
9321
2861
80
5113
18
19
141
92
7
75
95
0
203
32
9
323
4
168
81
2
102
91
5
102
12
32
690
10
86
62
15
64
60
96
8
639
20
3
173
13
08
142
17
44
101
16
57
98
13
79
165
Overall proportion = 39.2%Maximum = 65.5%Minimum = 9.33%Ratio, max:min = 7.02Chi square = 444.33 d.f. = 32 p = 0
Number "significantly" high 7Number "significantly" low 9Number "significantly" different 16
Getting data (4): all four forms
Services
%
02
04
06
0
% with all four forms5
32
300
14
96
315
10
87
51
13
85
69
2
932
44
8
55
5
164
30
08
135
1
286
13
89
128
18
23
40
10
40
153
4
168
20
21
77
7
430
13
62
49
38
7
69
18
05
1131
81
9
141
32
9
323
92
7
75
95
0
203
81
2
102
91
5
102
12
32
690
10
86
62
15
64
60
96
8
639
20
3
173
13
08
142
17
44
101
16
57
98
13
79
165
Overall proportion = 34.8%Maximum = 64.2%Minimum = 8.67%Ratio, max:min = 7.4Chi square = 611.05 d.f. = 32 p = 0
Number "significantly" high 13Number "significantly" low 9Number "significantly" different 22
Getting data: summaryFor each of these basic indices the differences
across services:were significant p<.0005were very large in magnitudethe number “significantly” different from overall
proportion ranged from 15 to 22 of the 33
Even at the “best” end, datasets are fairly incomplete …
… at the “worst” end completion rate is cripplingly low
Demographics (1): gender
Services
%
02
04
06
08
01
00
% Female2
02
1
77
38
7
69
4
168
30
08
135
32
9
323
44
8
55
18
05
113
12
32
690
20
3
173
95
0
2037
430
15
64
602
931
10
87
51
13
89
127
1
286
13
85
69
5
164
10
40
153
13
62
49
18
19
141
96
8
638
13
08
142
14
96
315
81
2
102
91
5
102
53
2
300
92
7
75
16
57
98
13
79
165
18
23
40
17
44
101
10
86
62
Overall proportion = 71.5%From n = 6607 n(miss) = 3 %(miss) = 0Higest proportion = 85.5%, lowest = 59.7%Ratio, max:min = 1.43Chi square = 63.53 d.f. = 32 p = 0.0008
Number "significantly" high 4Number "significantly" low 3Number "significantly" different 7
Demographics (2): ethnicity
Services
%
40
60
80
10
0
% White/European referrals1
08
7
51
32
9
303
13
89
107
12
32
661
17
44
99
1
224
92
7
74
14
96
3051
38
5
628
12
75
91
5
101
16
57
98
30
08
127
13
62
49
13
79
148
5
117
18
19
137
2
501
10
40
145
95
0
179
20
3
169
18
05
113
15
64
58
18
23
31
38
7
67
20
21
75
13
08
138
7
411
10
86
59
96
8
610
53
2
298
4
117
44
8
49
Overall proportion = 91.1%From n = 5758 n(miss) = 852 %(miss) = 13Higest proportion = 100%, lowest = 52.9%Ratio, max:min = 1.43Chi square = 686.72 d.f. = 32 p = 0
Number "significantly" high 11Number "significantly" low 5Number "significantly" different 16
Dotted red CI indicates %(miss) > 20%
Demographics (3): employment
Services
%
20
40
60
80
% full-time or part-time employed1
82
3
33
13
89
104
15
64
54
10
40
139
44
8
49
10
87
51
53
2
295
20
21
72
1
222
18
05
111
95
0
166
16
57
92
7
391
32
9
3041
30
8
1384111
14
96
301
13
85
62
96
8
598
91
5
96
81
2
94
12
32
664
13
62
48
92
7
75
18
19
134
2
420
20
3
168
13
79
134
17
44
99
38
7
67
10
86
56
5
115
30
08
126
Overall proportion = 55.6%From n = 5589 n(miss) = 1021 %(miss) = 15Higest proportion = 74.6%, lowest = 27.3%Ratio, max:min = 1.43Chi square = 188.05 d.f. = 32 p = 0
Number "significantly" high 8Number "significantly" low 9Number "significantly" different 17
Dotted red CI indicates %(miss) > 20%
Demographics (4): young age
Services
%
05
10
15
20
25
% under 20 years of age1
38
9
127
32
9
319
16
57
98
5
164
14
96
314
10
86
62
44
8
55
92
7
72
18
19
141
95
0
201
4
150
15
64
60
20
3
172
30
08
134
7
427
1
284
38
7
68
18
05
113
13
79
165
91
5
102
17
44
101
2925
18
23
401
23
2
687
13
85
69
81
2
102
96
8
633
53
2
299
10
40
152
10
87
51
20
21
77
13
62
49
13
08
141
Overall proportion = 4.5%From n = 6554 n(miss) = 56 %(miss) = 1Higest proportion = 10.6%, lowest = 0.787%Ratio, max:min = 1.43Chi square = 67.48 d.f. = 32 p = 0.0002
Number "significantly" high 1Number "significantly" low 4Number "significantly" different 5
Demographics (5): older age
Services
%
01
02
03
04
05
0
% over 59 years of age3
00
8
134
44
8
55
4
150
38
7
68
17
44
101
32
9
319
95
0
201
20
3
172
20
21
77
91
5
102
10
87
51
10
40
152
92
7
72
12
32
687
14
96
314
5
164
15
64
60
2
925
13
08
141
13
79
165
16
57
98
10
86
62
53
2299
18
23
401
80
5
113
18
19
141
81
2
102
96
8
633
7
427
1
284
13
85
69
13
62
49
13
89
127
Overall proportion = 6.47%From n = 6554 n(miss) = 56 %(miss) = 1Higest proportion = 38.6%, lowest = 0%Ratio, max:min = 1.43Chi square = 282.75 d.f. = 32 p = 0
Number "significantly" high 4Number "significantly" low 3Number "significantly" different 7
Demographics (6): age
Services
Site
me
dia
n
30
35
40
45
50
55
Median age9
27
72
20
21
77
2
925
10
87
51
12
32
687
4
150
5
164
20
3
172
32
9
319
38
7
68
53
2
299
91
5
102
13
08
141
17
44
101
30
08
134
10
40
152
1
2847
4274
48
55
95
0
201
96
8
633
10
86
62
13
79
165
14
96
314
15
64
60
16
57
98
18
05
113
18
19
141
18
23
40
13
85
69
81
2
102
13
62
49
13
89
127
Overall median = 37 on range from 11 to 89From n = 6554 n(miss) = 56 %(miss) = 1Highest median = 46 lowest = 33.5Ratio, max:min = 1.37Kruskal-Wallis chi square = 110.2 d.f. = 32 p = 0
Number "significantly" high 1Number "significantly" low 2Number "significantly" different 3
Demographics: summary
All differences p<.0005Quite large in magnitudeNumber “significantly” different from
overall proportion/median ranged from 3 to 16 of the 33
Particularly big differences on ethnicitySome of these demographic variables will
have relationships to outcome and failure both within and between services
Individual level & site level effects
Services
-1.0
-0.5
0.0
0.5
1.0
Sitewise correlations: age vs. CORE-OM non-risk improvement
128
45
74
2531
5984
16
35 37
98
311
22
61 67
30
139108
13
127
43
91
43
326339
57 4358
31
11
45
21
Overall correlation = 0.08From n = 2510 n(miss) = 4100 %(miss) = 62Maximum = 0.42Minimum = -0.4
Number "significantly" high 1Number "significantly" low 1Number "significantly" different 2
Dotted CI indicates %(miss) > 50%
Starting points (1): on medication
Services
%
20
40
60
80
% on psychotropic medication9
15
101
13
89
125
10
87
51
92
7
75
1
227
7
413
4
121
96
8
619
32
9
316
14
96
309
13
08
141
5
121
20
21
77
38
7
68
2
509
18
19
141
81
2
94
20
3
1731
36
2
493
00
8135
12
32
681
13
85
69
13
79
158
53
2
299
17
44
99
16
57
96
10
86
61
18
05
111
15
64
56
10
40
151
44
8
52
95
0
203
18
23
40
Overall proportion = 48.5%From n = 5941 n(miss) = 669 %(miss) = 10Higest proportion = 75%, lowest = 34.7%Ratio, max:min = -1.05Chi square = 141.29 d.f. = 32 p = 0
Number "significantly" high 6Number "significantly" low 5Number "significantly" different 11
Dotted red CI indicates %(miss) > 20%
Starting points (2): CORE-OM score
Services
Site
me
dia
n
1.5
2.0
2.5
Initial CORE-OM non-risk score1
38
9
107
10
87
20
13
85
62
91
5
99
10
86
60
20
3
164
18
05
100
53
2
246
14
96
291
18
19
132
92
7
71
30
08
100
96
8
5422
853
81
293
13
79
160
7
343
4
132
13
62
48
15
64
56
32
9
278
17
44
101
13
08
114
1
211
18
23
33
95
0
187
38
7
60
16
57
92
12
32
638
20
21
66
5
119
10
40
134
44
8
54
Overall median = 2.14 on range from 0.04 to 4From n = 5766 n(miss) = 844 %(miss) = 13Highest median = 2.39 lowest = 1.46Ratio, max:min = 1.64Kruskal-Wallis chi square = 123.63 d.f. = 32 p = 0
Number "significantly" high 4Number "significantly" low 4Number "significantly" different 8
Dotted CI indicates %(miss) > 20%
Starting points (3): % > CSC cut point
Services
%
40
50
60
70
80
90
10
0
% > CSC cut point1
38
9
107
20
3
164
13
85
62
53
2
246
10
86
60
10
87
20
13
62
48
15
64
56
96
8
542
81
2
93
13
79
160
91
5
99
4
1322
853
18
23
331
49
6
291
1
211
92
7
71
10
40
134
30
08
100
13
08
114
17
44
101
32
9
278
18
05
100
7
343
38
7
60
18
19
132
12
32
638
95
0
187
16
57
92
5
119
44
8
54
20
21
66
Overall proportion = 79.5%From n = 5766 n(miss) = 844 %(miss) = 13Higest proportion = 89.4%, lowest = 49.5%Ratio, max:min = 1.64Chi square = 107.65 d.f. = 32 p = 0
Number "significantly" high 2Number "significantly" low 3Number "significantly" different 5
Dotted CI indicates %(miss) > 20%
Starting points: summary
All statistically significant p<.0005 Large differences Number “significantly” different from overall
proportion/median ranged from 6 to 10 of the 33 Again, starting conditions can have relationships with
outcome and failures at both individual and service level
Logistics (1): wait time to assessment
Services
Site
me
dia
n
05
01
00
15
0
Wait time to assessment4
48
3
53
2
77
20
21
76
10
87
47
12
32
612
13
79
149
13
89
113
5
94
2
378
13
08
101
18
19
72
10
40
118
1
177
7
399
92
7
63
18
23
39
30
08
128
18
05
113
96
8
418
91
5
581
08
6
611
36
217
38
7
64
95
0
198
4
89
17
44
101
32
9
317
16
57
90
15
64
58
14
96
216
20
3
168
13
85
26
Overall median = 31 on range from 0 to 583From n = 4640 n(miss) = 1970 %(miss) = 30Highest median = 137 lowest = 0Ratio, max:min = Inf.Kruskal-Wallis chi square = NA d.f. = 32 p = NA
Number "significantly" high 8Number "significantly" low 11Number "significantly" different 19
Dotted CI indicates %(miss) > 20%
Logistics (2): % offered more sessions
Services
%
50
60
70
80
90
10
0
% offered more sessions1
38
9
128
18
23
40
10
40
153
30
08
135
10
87
51
13
85
69
44
8
55
18
19
141
1
232
53
2
300
91
5
102
10
86
62
2521
7
425
81
2
102
4
124
92
7
75
32
9
323
95
0
203
12
32
690
96
8
639
14
96
315
20
3
173
13
79
165
20
21
77
17
44
101
18
05
113
38
7
69
13
08
142
16
57
98
13
62
49
5
121
15
64
60
Overall proportion = 88.4%From n = 6053 n(miss) = 557 %(miss) = 8Higest proportion = 100%, lowest = 68.8%Ratio, max:min = Inf.Chi square = 248.69 d.f. = 32 p = 0
Number "significantly" high 6Number "significantly" low 7Number "significantly" different 13
Dotted CI indicates %(miss) > 20%
Logistics (3): #(sessions planned)
Services
Site
me
dia
n
24
68
10
Number of sessions planned4
48
24
53
2
151
20
21
65
13
62
14
1
145
2
269
4
79
5
94
7
238
20
3
163
38
7
43
81
2
52
91
5
819
27
519
50
147
96
8
391
10
40
64
10
87
341
23
2
554
13
08
74
13
79
125
13
89
54
14
96
80
16
57
86
17
44
91
18
05
60
18
19
121
18
23
18
30
08
71
10
86
50
13
85
32
15
64
39
32
9
270
Overall median = 6 on range from 0 to 60From n = 3830 n(miss) = 2780 %(miss) = 42Highest median = 10 lowest = 3Ratio, max:min = 3.33Kruskal-Wallis chi square = 1036.41 d.f. = 32 p = 0
Number "significantly" high 4Number "significantly" low 2Number "significantly" different 6
Dotted CI indicates %(miss) > 20%
Logistics: summary
All p<.0005 All large differences, particularly for waiting time from
referral to assessment (13 days cf. 137 days) Number “significantly” different from overall
proportion/median ranged from 6 to 19 of the 33 There are big differences on number of sessions offered
(medians from 3 to 10) … but many services offering fixed number, mode is six
sessions Looks very likely that there will be some differences
between services in the ways they operate that will hugely affect outcome and failures
Outcomes (1): unplanned endings
Services
%
02
04
06
08
01
00
% offered more sessions1
37
9
133
15
64
39
91
5
82
96
8
533
4
102
13
89
67
81
2
76
7
240
16
57
85
92
7
56
32
9
260
13
08
122
10
87
37
5
99
17
44
93
20
3
166
1169
13
62
261
08
649
38
7
44
12
32
585
18
19
125
18
05
77
10
40
90
30
08
69
2
278
95
0
178
14
96
90
18
23
20
53
2
279
20
21
70
13
85
25
44
8
32
Overall proportion = 38.5%From n = 4396 n(miss) = 2214 %(miss) = 33Higest proportion = 65.6%, lowest = 17.3%Ratio, max:min = 3.33Chi square = 162.78 d.f. = 32 p = 0
Number "significantly" high 6Number "significantly" low 4Number "significantly" different 10
Dotted CI indicates %(miss) > 20%
Outcomes (2): CORE-OM change
Outcomes (2): CORE-OM change
Services
Site
me
dia
n
-3-2
-10
Change on CORE-OM non-risk score
5
59
10
87
8
12
32
312
13
08
74
2
340
53
2
25
4
72
18
05
43
1
98
92
7
31
44
8
12
7139
81
243
20
391
96
8
326
13
85
13
16
57
61
17
44
58
95
0
84
38
7
22
32
9
129
18
23
11
30
08
37
13
79
108
18
19
57
20
21
21
10
40
43
13
62
16
91
5
45
10
86
31
13
89
35
15
64
30
14
96
45
Overall median = -1.09 on range from -3.68 to 2.18From n = 2519 n(miss) = 4091 %(miss) = 62Highest median = -0.5 lowest = -1.39Ratio, max:min = 0.36Kruskal-Wallis chi square = 112.44 d.f. = 32 p = 0
Number "significantly" high 5Number "significantly" low 3Number "significantly" different 8
Dotted CI indicates %(miss) > 50%
Outcomes (3): % RC
Services
%
40
60
80
10
0
% with reliable improvement1
49
6
45
15
64
30
13
89
35
91
5
45
10
40
43
32
9
129
13
85
13
30
08
37
13
79
108
10
86
31
81
2
43
38
7
22
10
87
8
13
62
16
16
57
611
81
957
1
98
95
0
84
96
8
327
20
3
91
7
139
20
21
21
17
44
58
18
05
43
12
32
312
4
72
2
340
92
7
31
13
08
74
53
2
25
5
59
18
23
11
44
8
12
Overall proportion = 78.4%From n = 2520 n(miss) = 4090 %(miss) = 62Higest proportion = 100%, lowest = 53.3%Ratio, max:min = 0.36Chi square = 76.97 d.f. = 32 p = 0
Number "significantly" high 3Number "significantly" low 4Number "significantly" different 7
Dotted CI indicates %(miss) > 50%
Outcomes (4): % CSC
Services
%
20
40
60
80
10
0
% with clinically significant improvement1
08
6
31
13
89
35
20
21
21
15
64
30
13
62
16
13
85
13
10
40
43
14
96
45
32
9
129
91
5
45
81
2
43
4
72
16
57
61
20
3
91
13
79
108
30
08
37
38
7
22
18
23
11
96
8
327
44
8
121
98
95
0
84
53
2
25
18
19
57
12
32
312
17
44
58
7
139
2
340
18
05
43
92
7
31
13
08
74
5
59
10
87
8
Overall proportion = 58.8%From n = 2520 n(miss) = 4090 %(miss) = 62Higest proportion = 87.5%, lowest = 32.3%Ratio, max:min = 0.36Chi square = 63.72 d.f. = 32 p = 0.0007
Number "significantly" high 2Number "significantly" low 2Number "significantly" different 4
Dotted CI indicates %(miss) > 50%
Outcomes: summary
All statistically significant p<.0005Large differencesNumber “significantly” different from
overall proportion/median ranged from 4 to 9
Despite large differences on RC and CSC, number of services differing “significantly” from the overall is not so high (4 and 6 respectively)
Can automation of data processing help bridge the gap?
Neither researchers nor practitioners know much about the generalisability of “strong causal inference” to routine practice
Need practice to come out of the confidentiality closet without harming true confidentiality
Very, very few services currently collect routine outcome data
Few services link with other services to compare practices and data
Few services have strong links to researchers to help understand data
Need to bridge these gaps: if we make data easier to handle it might help!
Automation (1): batch routeFacilitates some distancing from the dataData analyses done by researchers and
experts in analysis and data handlingReports (30+ pages) well receivedCan explore site specific issues
Automation (2): CORE-PC
“The clinical and reliable change graph is invaluable. As a service manager it gives me instant access to where we can look to improve our service provision”
Automation (2): CORE-PC
“I never realised that writing a report could be so simple, all I need to do is copy the tables I need from CORE-PC, paste them in Word, and write my interpretations.”
Automation (2): PC
Allows services to get much “nearer” to their data
Should prevent some data entry errorsShould increase data completenessMay mean that service clinicians and
managers feel uncertain about how to analyse and interpret their data…
… will need training and support
http://www.psyctc.org/stats/Weimar
Not until Monday 30.vi.03!