a better start evaluation jane barlow university of warwick

A Better StartEvaluation

Jane BarlowUniversity of Warwick

Who are we?

http://www.ipsos-mori.co.uk/

What are we aiming to do?

Conduct of a robust evaluation of A Better Start which will run throughout the 11-year period of the programme:Focus on the setup, implementation and

impact of the programme within and across the areas

Disseminate learning across the areas involved and more widely

How do we plan to do it?

• Workstream 1: Implementation evaluation of the setup and delivery of the programme;

• Workstream 2: Impact and economic evaluation of the area programmes;

• Workstream 3: A programme of learning and dissemination that will extend across the five areas, and beyond

Workstream 1:Implementation

Evaluation

Mairi Ann Cullen CEDAR, University of

Warwick

Outline

• The team• Aims and design• Phase 1

–First steps–Rest of phase 1

• Outline of Phase 2

The Team

• CEDAR (Centre for Educational Development, Appraisal and Research)

• Research– Geoff Lindsay, Mairi Ann Cullen,

Stephen Cullen• Admin

– Diana Smith, Alison Baker, Shauna Yardley

Aims

• To evaluate the setup and delivery of A Better Start–Each area as a case study–Programme as a whole

• And developments outward, across other areas of England, and beyond

Our approach

• Participatory and collaborative research– The research team, the areas and BIG– Participatory action approach

• Working with the areas to understand each area’s programme

• Examining the setup• Create the evaluation for Phase 2

• Independent evaluation• Responsibilities: CEDAR, Areas, BIG

Participatory action

approach

Independent evaluation

Responsibilities to optimise evaluation

Methods• Ethical approval – University of Warwick• Data collected by us:

– Interviews, surveys, document analysis• Data collected by you

– Reviewing your monitoring data, reports etc

Methods

Phase 1 setup (now – end 2015)

• Working with areas to understand your programmes and evaluate the implementation of the setup phase– Feeding back to enable learning optimising each area’s programme

• Common and area specific objectives– Co-production

• Context/Inputs, activities, outputs

Where are we now? • Context and Inputs, e.g.• Identification and mapping of current

services, interventions, delivery mechanisms, data monitoring, and reporting to create baseline scenarios, to include e.g. – Needs analysis – Infrastructure including staff, IT systems,

management systems – Dartington’s support

(cont.)• Development and agreement of policies and

procedures for A Better Start (e.g. examination of evidence for possible interventions and decision-making regarding the selection): – Processes for agreeing the interventions – Infrastructure to implement and manage

the interventions– Data monitoring– Budget creation – Pressures

Activities• Process implementation – putting the

agreed policies and procedures into place e.g.: –Staff recruitment –Training –Supervision –Data collection and management

to track progress–Financial system

Outputs• What are the results?[NB this will largely occur at Stage 2 when the interventions are underway] • E.g. locally collected pre- and post-

intervention measures–E.g. improvements in children’s

language and communication• Common measures where possible

Examples– Priorities within the ABS framework– Selection of interventions– Service configuration– Staffing and training– Governance system– Management system– Data collection, analysis and

reporting system– Parents’ (and others’) engagement

First steps• Initial discussions here • Where are you at?

– Your timetable of activities , e.g.• Recruitment of key staff, • Governance set up and meetings.• Selection of interventions

– Arrange visits to August – September?• Interviews with strategic and operational

leads• Identify other interviewees, e.g. heads of

services, specific programmes

October – end 2015

• Fieldwork in each site• Common data for each area

– To enable aggregation across 5 areas e.g. largely standard interview schedules with key personnel

– To enable comparison, e.g. different approaches to same objective

– Specific data for each area

October – end 2015• Timetable

– Constructed with each area to meet final end of Phase 1 deadline agreed with BIG

– Activity moving from exploiting the setting up of the systems to evaluating the systems in action, e.g.• What supports optimal implementation• What are the barriers?• How can this learning be used to revise

systems• And to inform others?

Phase 2• Starts 2016 – subject to progress• 5 year study of the areas’ systems in

operation, including:– Interviews and surveys of personnel,

stakeholders, users (Process).– Examination of locally collected data

(Outputs and Outcomes),• E.g. comparisons of pre-post data

from interventions to examine change

Workstream 2: Impact & Economic Evaluation

Impact studyOverview of surveys

Sally Panayiotou Research Director, Ipsos MORI

http://www.ipsos-mori.co.uk/

How will this be carried out?• We will track two cohorts (an early and a late

cohort) of families in the service areas in addition to matched cohorts of similar families living in carefully selected comparison areas

• Collect i) individual; and ii) population level data for 3 key outcomes (nutrition; language; socioemotional development)

• Following families from pregnancy to 7 years

What is the purpose of the surveys?

Identify short- medium- and long-term changes in:

parental functioning children

Socio-emotional

health

NutritionSpeech,

language and

learning

How will we measure these changes?

• Warwick consortium composed of specialists in each of these fields

• Series of surveys starting in pregnancy• Range of validated measures and survey questions

DepressionEdinburgh Post-

Natal Depression Scale (EPDS)

Bonding with Baby

Mother-to-Infant-Bonding Scale

(MIBS)

AnxietyState-Trait

Anxiety Inventory

Brief Infant-Toddler social and emotional

assessment (BITSEA)

Patterns of food intake

Children’s Dietary

Questionnaire (CDQ)

Child’s cognitive

abilityBritish Ability Scales (BAS)

For example

Additional biometric measuresSome participants in the study asked if they would be happy for their child to also participate in additional biometric measures

Hair Sample Age 2

Measure stress hormone, cortisol

Buccal (cheek) swab Age 2

Measure epigenetic changes (i.e. which inherited genes are

‘switched on or off’ due to environment)

Interviewer team

Measures suggested below will only be requested from approx. 10% of participants, IF:

• National research ethical approval is granted, and;

• Intervention areas approve, and;• After consultation with local ‘user groups’ to

seek their opinions, and address any concerns

Who will we be speaking to?• Mothers recruited to the survey at 16

weeks in their pregnancy – series of face-to-face and postal interviews

• Paper questionnaires for the partners where applicable

• During the course of the evaluation we follow the child

Where will we be conducting interviews?

Interviews in: 5 selected areas =

programme sample 15 matched comparison areas =

comparison sample

Baseline studyGain a pre-intervention measure of outcomes in 2015

Initial pilot of 90 interviews to check survey materials

1620 face-to-face interviews with families across the programme and comparison areas. Interviews with :

Mothers of 1 year-olds

Mothers of 2 year-olds

Mothers of 3 year-olds and survey tasks with the child

Two cohort studiesCohort 1 – starts in the second year of

the evaluation (2016)

Cohort 2 – starts in the fourth year of the

evaluation (2018)

Programme n= 775

(150 per area)

Matched comparison n= 550 (35 per area)

Programme n= 1710

(340 per area)

Matched comparison n= 1170 (75 per area)

(Numbers are approximate)

What will recruitment involve?

• Led by Debra Bick and Sarah Beake, Florence NightingaleSchool of Nursing and Midwifery

• Kings Health Partners will identify tertiary maternity units in the 5 intervention and 15 comparisonareas

• Contacts/meetings with the Headsof Midwifery in each unit. Letter ofsupport from units which agree totake part

Recruitment Process• All pregnant women will be sent a study information leaflet

with their pregnancy booking information from before their

first antenatal appointment at 12 weeks • At their 16 week routine antenatal appointment, the midwife

will:- Check women received a study information leaflet and offer another leaflet if appropriate

- Ask women if they do not want their contact details forwarded to the research team

• Ipsos MORI team will contact the women to arrange date to meet when women 24-32 weeks gestation

• At this first interview, women will be asked for written consent to participate in the study

• Midwives will update the team about women no longer eligiblefor inclusion (i.e. if pregnancy loss)

Survey pointsSurvey point Survey with mother (/ main carer in later stages

if child does not live with mother)

1 24-32 weeks pregnant Face-to-face in-home

2 Baby aged 2-months Postal

3 Baby aged 4-months Telephone

4 Child aged 1 Face-to-face in-home



7 Child aged 5 Postal / online

8 Child aged 7 (Cohort 1 only) Postal / online

Survey pointsSurvey point Mother / main carer Partner paper

questionnaire









Additional measures with a sub-sample of participants

Survey point Mother / main carer Additional assessments




4 Child aged 1 Face-to-face in-home CARE Index (video-coding of free play)

5 Child aged 2 Face-to-face in-home Cheek swab (epigenetic changes)Hair sample (cortisol)

6 Child aged 3 Face-to-face in-home Attachment story stem



Looking after participants during the study

Interviews will be conducted by experienced Ipsos MORI interviewers in line with the MRS code of conduct – they will receive full training on this study

The study will gain ethics approval, and local R&D approvals from all study sites prior to sending women any information

We will gain informed consent at each survey stage Participants will be free to opt-out of taking part at any stage All survey data will be anonymised We will provide a supporting participant website, summaries

during the research and information leaflets with details on where to seek advice on the issues covered in the survey

We will provide respondents with a survey email and a free-phone telephone number for them to contact the survey team at any stage

If survey responses indicate a participant is at serious risk of harm we will seek their permission to contact an appropriate service provider on their behalf; if permission is not given we will still advise them to seek support

How could you help us with recruitment?

• What contacts do you already have with your local maternity units?

• Are you aware of any ongoing recruitment of women in early pregnancy to studies in your area?

• Could you advise on maternity units in your area that we should contact in the first instance?

What do you need to do during the surveys?

All survey work will be carried out by Ipsos MORI and there is nothing you need to do during the surveys

But we would really

appreciate your support!

During the surveys if you receive any queries please contact us and we will respond quickly to resolve any issues as soon as we

can

Will you receive the anonymised survey data?

Yes – we would like to discuss the best format for you in the context of the Management Information

System

Workstream 3: Dissemination and Learning

Jane BarlowUniversity of Warwick

What does this mean?

• Essential that ongoing & final findings from the evaluation be shared:

- wider community- local authorities- CCGs- third sector providers- central government- policy interest organisations

Why is this important?

• Sharing learning from the evaluation will help to:

- improve practice- influence local and national decision-makers to make fundamental shift in policy in early years prevention

How will this be done?• Exact methodology not yet completely defined,BUT, likely to involve:- supporting peer learning within and across intervention areas- ‘Learning & Dissemination’ website/online evaluation resource & forum- delivering effective learning amongst 5 intervention areas to ensure ongoing improvement in interventions, approaches and systems change

Conclusions

• ABS is a tremendous opportunity to improve children’s lives

• And to improve our knowledge of effectiveness of interventions– What works? For whom? Under what

circumstances? And what aids or undermines its success.

– We look forward to working with you!!

44

a better start evaluation jane barlow university of warwick

Documents