Assessment of Intervention Outcomes

Workshop: Thursday 7th FebruaryIYW Conference, Llandrindod Wells

Assessment of Intervention Outcomes

Workshop: Thursday 7th FebruaryIYW Conference, Llandrindod Wells

Dr Dave Daley & Dr Tracey BywaterDr Dave Daley & Dr Tracey Bywater

QuickTime™ and aPhoto - JPEG decompressor

are needed to see this picture.

2

Overview of workshopOverview of workshop

Rationale for conducting your own evaluations

How to manage the numbers

Suggested evaluation material

Rationale for conducting your own evaluations

How to manage the numbers

Suggested evaluation material

3

Why evaluate clinical work?Why evaluate clinical work?

NHS Trusts need require you to conduct evaluation

It is imperative to be able to demonstrate improvement associated with your groups

Helps to maintain funding and/or win new funding

NHS Trusts need require you to conduct evaluation

It is imperative to be able to demonstrate improvement associated with your groups

Helps to maintain funding and/or win new funding

4

Is it difficult to evaluate outcome?

Is it difficult to evaluate outcome?

The most important decisions to make when considering evaluation are:

1) Design

2) Evaluation measures

The most important decisions to make when considering evaluation are:

1) Design

2) Evaluation measures

5

1. Design1. Design

It is very important that you have baseline measures (before) and outcome measures (after)

It is also important that you have the same measures on everyone

It is very important that you have baseline measures (before) and outcome measures (after)

It is also important that you have the same measures on everyone

6

What can we do with our evaluation scores I?

What can we do with our evaluation scores I?

Even simple spreadsheet programmes like Excel will allow you to conduct simple statistics

For evaluation purposes you are most interested in change from start to end

Even simple spreadsheet programmes like Excel will allow you to conduct simple statistics

For evaluation purposes you are most interested in change from start to end

7

Pros & Cons of change scoresPros & Cons of change scores

Change scores are most useful Easy and simple way of evaluating change However recent suggestion that change

scores on SDQ are less sensitive However formula is complicated, has not

yet been fully validated and is not recommended for evaluations of small groups

Change scores are most useful Easy and simple way of evaluating change However recent suggestion that change

scores on SDQ are less sensitive However formula is complicated, has not

yet been fully validated and is not recommended for evaluations of small groups

8

What can we do with our evaluation scores II?

What can we do with our evaluation scores II?

Change scores should demonstrate improvements in child behaviour outcome

You can also begin to examine whether your groups are more effective for particular types of parents

Change scores should demonstrate improvements in child behaviour outcome

You can also begin to examine whether your groups are more effective for particular types of parents

9

What can we do with our evaluation scores III?

What can we do with our evaluation scores III?

Rather than do lots of fancy statistics, it is also interesting to examine scores at outcome

Most evaluation measures will have scores which indicate clinical concern

It is very easy but also meaningful to examine the percentage of children and/or parents who score above cut off scores before intervention and after intervention

Rather than do lots of fancy statistics, it is also interesting to examine scores at outcome

Most evaluation measures will have scores which indicate clinical concern

It is very easy but also meaningful to examine the percentage of children and/or parents who score above cut off scores before intervention and after intervention

10

2. Evaluation Measures2. Evaluation Measures

Which ones? Pathfinders project uses:

Demographic Questionnaire Beck Depression Inventory Eyberg Child Behaviour Inventory Strengths & Difficulties Questionnaire x 2 Parenting Scale Overt Hostility Scale Teacher Questionnaire/Report

Which ones? Pathfinders project uses:

Demographic Questionnaire Beck Depression Inventory Eyberg Child Behaviour Inventory Strengths & Difficulties Questionnaire x 2 Parenting Scale Overt Hostility Scale Teacher Questionnaire/Report

11

DemographicsDemographics

What do you need to know? Do you want to compare outcomes of:-

older versus younger parents? Boys versus girls? Different areas? Any other ideas?

What do you need to know? Do you want to compare outcomes of:-

older versus younger parents? Boys versus girls? Different areas? Any other ideas?

12

The Eyberg Child Behaviour Inventory

The Eyberg Child Behaviour Inventory

36 items, ages 2-16 years, 10 minutes to complete

Problem score has a minimum of 0, maximum of 36 (yes/no answers, problem or not)

Intensity scale - minimum score of 36, maximum 252 (scale of 1 – 7, where 1 = never and 7 = always)

Clinical cut-off scores, ≥127 intensity, or ≥11 problem

Purchase from http://www3.parinc.com/

36 items, ages 2-16 years, 10 minutes to complete

Problem score has a minimum of 0, maximum of 36 (yes/no answers, problem or not)

Intensity scale - minimum score of 36, maximum 252 (scale of 1 – 7, where 1 = never and 7 = always)

Clinical cut-off scores, ≥127 intensity, or ≥11 problem

Purchase from http://www3.parinc.com/

13

Strengths & Difficulties Questionnaire

Strengths & Difficulties Questionnaire

FREE, website: http://www.sdqinfo.com/b1.html 10 minutes to complete, 25 items:

Emotional symptoms Conduct problems Hyperactivity Peer relationship problems Prosocial behaviour An additional impact supplement

Parent & teacher Normal, borderline & atypical ranges/cut-off

FREE, website: http://www.sdqinfo.com/b1.html 10 minutes to complete, 25 items:

Emotional symptoms Conduct problems Hyperactivity Peer relationship problems Prosocial behaviour An additional impact supplement

Parent & teacher Normal, borderline & atypical ranges/cut-off

14

Beck Depression InventoryBeck Depression Inventory

21-item elf-report inventory measuring the severity of characteristic attitudes & symptoms associated with depression

Each item contains four possible responses which range in severity from 0 ( I do not feel sad) to 3 ( I am so sad or unhappy that I can’t stand it)

Score of 10-18 = mild to moderate depression Score of 19-29 = moderate to severe depression

21-item elf-report inventory measuring the severity of characteristic attitudes & symptoms associated with depression

Each item contains four possible responses which range in severity from 0 ( I do not feel sad) to 3 ( I am so sad or unhappy that I can’t stand it)

Score of 10-18 = mild to moderate depression Score of 19-29 = moderate to severe depression

15

Statistics with little mathsStatistics with little maths

Excel Graphs

Mean & SD Change scores Effect sizes

Excel Graphs

Mean & SD Change scores Effect sizes

16

Welsh Sure Start Data Welsh Sure Start Data

before SD after SD change scoresBDI-I 17.74 10.84 10.33 10.15 7.41BDI-C 15.04 9.68 13.66 10.31 1.38ECBI-PI 16.56 6.92 9.55 7.94 7.01ECBI-PC 14.75 7.69 13.68 8.27 1.07ECBI-II 145.92 25.19 116.19 32.09 29.73ECBI-IC 140.11 25.64 142.98 32.41 2.87

17

Effect sizesEffect sizes

Intervenfup1

Controlfup1

Difference pooled SD

effect sizes

BDI 10.33 13.66 3.33 10.23 0.33

ECBI-P 9.55 13.68 4.13 8.11 0.51

ECBI-I 116.19 142.98 26.79 32.25 0.83

Cohen’s 1988 guidelines: difference between means divided by pooled SD. 0.3 = clinically useful change, 0.5 medium effect, 0.8 = large effect

18

Any questions?????Any questions?????