commissioning and working with outcomes

Commissioning and Working with Outcomes

Measuring Outcomes and Demonstrating Impact in Schools

Mick Atkinson – Head of Commissioning, Place2Be

Key questions

• What is the mission of your organisation?

• Why does your organisation measure outcomes?

• Which outcomes are you trying to achieve?

• Do your measurement tools fit with the above three questions?

Place2Be

“Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted”

Albert Einstein

From mission to impact

To enhance the wellbeing and prospects of children and their families by providing access to therapeutic and emotional support in

schools, using a proven model backed up by research and training.

Inputs →→→→ Outputs →→→→ Outcomes →→→→ Impact

• Staff • One-to-one

support

• Decrease in children’s social and

emotional difficulties

• Positive emotional wellbeing /

intelligence

• Budget • Group work

support

• Improved educational attainment

• Safer and stronger

communities

• Dedicated room

in school

• Lunchtime ‘drop-in’

service

• Improved attendance rates

− Reduction in crime and

anti-social behaviour

• Multi-agency

networking

• Interventions for

parents

• Reduced truancy and exclusion

rates

− Increased skilled

workforce

• Training for school

staff

• Improved mental health of parents

− Economically active and

financially secure society

• Support teachers

with Circle Time

• Improved quality of parent/child

relationships

• Enhanced emotional intelligence

and skill-base of school staff

• Increased whole-school emotional

awareness/ intelligence

Clinical Outcomes in Routine Evaluation (CORE-OM)1

• Recommended by National Family Parenting Institute

• 34-item questionnaire designed to measure parents’ global level of distress

• Taps into a pan-theoretical ‘core’ of clients’ distress, including subjective wellbeing, commonly experienced problems or symptoms,and life/social functioning. In addition, items on risk to self and others are included

• Can be compared with clinical thresholds before and after intervention to determine clinically significant change

• Externally validated and normative data exists enabling us to compare outcomes with those from external clinical and non-clinical populations 1www.coreims.co.uk

CORE-OM

Scoring the CORE-OM

Each item within the CORE-OM is scored on a 5-point scale ranging from 0 to 4. The total score is calculated by adding the response values of all 10 items. The minimum score that can be achieved is 0 and the maximum 40.

Clinical and risk categories

Scores indicate the following clinical categories:0 – 5 Healthy

6 – 10 Low level

11 – 15 Mild

16 – 20 Moderate

21 – 25 Moderate severe

Over 25 Severe

The ‘risk’ cut-off score is 3

Parents’ Total ‘Global Distress’ (including Risk) scores pre-and post-intervention

• 96% (82) of the 85 parents had lower Total Global Distress scores following intervention. • The proportion of parents who scored within the ‘Clinical’ range (indicating mild to severe levels of Global Distress)

reduced from 82% (70 parents) to 22% (19 parents) over the intervention period.

• In total, of the 70 parents who scored within the ‘Clinical’ range prior to intervention, 74% (52 parents) achieved

‘Clinical recovery’; thus moving into the Non-clinical range following intervention.

YP CORE

What is the YP CORE?

• The Young Persons Clinical Outcomes in Routine Evaluation (YP CORE) is a standardised 10 item measure commonly used by counselling services for evaluation and outcome measurement of 11 to 16-year-olds.

• The YP CORE is short, therefore not too onerous, and focuses on feelings that young people are experiencing.

• The measure has eight negative and two positive items. It includes a single (negatively framed) risk-to-self item.

• Each item is scored from 0 to 4 on a Likert scale, with lower scores indicating lower levels of psychological distress.

Children's Outcomes Rating Scale (CORS) (1)

Description

• Developed by the Institute for the Study of Therapeutic Change (ISTC).

• Consists of a four-item visual outcome scale for children assessing areas of life functioning (individual, family, school, everything) to be completed at the beginning of every session.

• This questionnaire is designed for use throughout counselling and at its closure to provide ongoing information as to how the child is feeling in order to inform the Practitioner’s decision-making and in theory, to enhance treatment effects.

• There is also a session measure to assess the child’s experience of the service. The intention is that client complaints can be raised and addressed during therapy and sessions tailored to the client’s individual needs.

Children's Outcomes Rating Scale (CORS) (2)

Age suitability

6-12

There is an even more simplified version of the outcome and session measure for use with very young children where they are asked to select a face (e.g. happy / sad / angry) that best describes how they feel in the last week / about the session.

About the Strengths and Difficulties Questionnaire (SDQ) 1 (1)

What is the SDQ?

• The SDQ is a widely used behavioural screening and assessment questionnaire for children between the ages of 3 and 16. The tool was recommended to Place2Be by the Department of Health, and we have been using it in the evaluation of our one-to-one counselling interventions since 2001.

1www.sdqinfo.com

About the Strengths and Difficulties Questionnaire (SDQ) 1 (2)

Who completes the SDQ?

• There are separate SDQs to be completed by the teacher, the parent, and the child.

• The SDQ is completed at two time points: pre-intervention and post-intervention. The pre-intervention scores are compared with the post-intervention scores to see whether there has been any change and in which direction the change lies. The measure of change helps to show how effective the work with the child has been.

1www.sdqinfo.com

What does the SDQ measure? (1)

The SDQ

The SDQ consists of 25 behavioural statements. The 25 items are grouped into five psychological attributes, four of which represent negative aspects of the child’s psychological attributes, four of which represent negative aspects of the child’s behaviour (Emotional Symptoms, Conduct Problems, Hyperactivity, Peer Problems; the ‘Difficulties’scale). The fifth scale represents the child’s Strengths i.e. the child’s positive behavioural qualities (‘Prosocial’ scale).

The four Difficulties sub-scales can be summed together to form a ‘Total Difficulties’ score that ranges from 0-40. These scores can be classified into clinical bandings:

Normal Borderline Abnormal

Teacher 0-11 12-15 16-40

Parent 0-13 14-16 17-40

Child 0-15 16-19 20-40


The SDQ impact supplement

On the back of the questionnaire there are additional questions relating to the degree to which the child’s problems have an impact on different areas of their life. This is called the ‘impact supplement’. The items in the impact supplement can be summed to generate an impact score that ranges from 0 to 6 for the teacher-completed version and 0 to 10 for the parent- and self-report version.

Normal Borderline Abnormal

Teacher 0 1 2-6

Parent and child 0 1 2-10


• Using the clinical bandings, an abnormal score can be used to identify likely ‘cases’ with mental health disorders.

SDQ outcomes for Place2Be 1:1 interventions 2010/11 (1)

Percentage (and number) of children who showed improvement in Total Difficulties scores post-intervention, by informant

67%

(1,112)

72%

(883)70%

(840)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Teacher Parent Child


Percentage (and number) of pre-intervention Abnormal clinical category

children who showed improvement in Total Difficulties scores post-intervention, by informant

79%

(690)

82%

(532)

86%

(357)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Teacher Parent Child


Question: “Since coming to [Place2Be], how are this child’s/your problems?”

1%

(18)

2%

(19)

1%

(20)

3%

(33)

3%

(32)

4%

(69)

9%

(113)

14%

(170)

27% (445)

31% (384)

36% (446)

42% (682)

56% (690)

47% (582)

27% (433)

0% 20% 40% 60% 80% 100%

Teacher-report

Parent-report

Self-report

SD

Q info

rmant

Percentage (and number) of children

Much worse

A bit worse

About the same

A bit better

Much better

How appropriate is the SDQ in demonstrating outcomes?

Advantages:

• Externally validated

• Reliable

• Gathers opinion from more than one perspective

• Can compare outcomes to normative data

• Used by CAMHS providers

• Recommended by C&YP-IAPT and CORC

• Available in over 50 languages

Challenges:

• Variance between informants’ ratings

• Informant bias

• Not valid for children under 11

• Sensitivity to the needs of the children we work with

• Bias: SPMs as the Supervisors and evaluators of 1:1 and group work

Why is Place2Be collecting dataon children’s educational outcomes?

• To find out if improving children’s wellbeing influences

children’s learning.

• By enabling children to better manage and regulate their

feelings and develop their relationships skills, children may be

more able to focus in class and work well with their peers and

teachers.

• However, a huge range of factors affect children’s academic

progress including their prior learning, home environment, SEN

and individual ability.

What data is Place2Be collecting on children’s educational outcomes?

Data on children who start counselling in Years 3 to 9 (not Early

Years currently)

• National Curriculum sub-levels

• Pupil progress - whether teachers believe students are on

track with their learning

• Children’s attitude to learning (primary schools only)

• Attendance (secondary schools only)

National curriculum sub-levels

• 3 time points: the summer before the start of counselling, the

summer after the start of counselling and two summers after

the start of counselling.

• In primary schools, Reading, Writing and Maths sub-levels. In

secondary schools, English, Maths and Science.

• Each Level is made up of three sub-levels. Most pupils are

expected to progress one Level every two years (equivalent to

1.5 sub-levels over one year).

Children’s attitudes to learning andpupil progress

• Teacher’s report on the past 2 months in school, pre- and post-

intervention.

• Attitude to learning: concentrating hard in lessons, enjoys

classroom work, likes reading, likes writing and likes maths.

(Primary only)

• Pupil progress for Reading/Writing/Maths in primary schools and

English/Maths/Science in secondary schools.

• Whether child is doing better than/as well as/worse than

expected.

Pilot study of outcomes for secondary school children

• Small opportunity sample. English and Maths NC data was collected

for summer before and summer after start of counselling for 40

children receiving one-to-one counselling in two Harlow schools.

Complete and clean data was only available for 28 children.

• Progress of Place2Be sample compared to large nationally

representative sample of children in Years 7 to 9. Our sample had a

much higher level of need:

• 37% FSM compared to 17% in national sample

• 72% SEN in comparison to 33% in national sample

GETTING IN TOUCH

www.youngminds.org.uk/bond

[email protected]

020 7089 5050

www.place2be.org.uk

[email protected]

020 7923 5500

commissioning and working with outcomes

Documents