transforming education through evidence

TRANSFORMING EDUCATION THROUGH EVIDENCE.

The Centre forEffective Education

SCHOOL OFEducation

Conducting Educational Randomised Control Trials in Disadvantaged Community Settings

Andy Biggart & Liam O’Hare

SCHOOL OF Medicine, Dentistry and Biomedical Sciences

Presentation Overview1. Introduction to research context and

programmes

2. Overview of Key Results

3. Conextualising Results in Wider Literature

4. Some methodological and ethical reflections of conducting RCT’s in community settings

The Centre for Effective Education | SCHOOL OF Education

SCHOOL OF Medicine, Dentistry and Biomedical SciencesThe Centre for Effective Education | SCHOOL OF Education

Research Context• Educational Research & Randomised Control

Trials

• The Atlantic Philanthropies

• Some of first contemporary trials in UK and Ireland

• The two progammes grew out of a consultation with the local community in Tallaght (Dublin)


Two After-schools Programmes• Doodle Den

• Newly developed after-schools programme

• Aimed at 5 and 6 year olds identified as struggling readers

• Adopted a balanced literacy framework

• Delivered by teacher and youth worker/child care assistant

• Ran throughout the school year

• 3 days a week for 1.5 hours afterschool

• Highly structured programme with strong emphasis on professional development


Two After-schools Programmes• Mate Tricks

• Very similar in delivery to Doodle Den

• Hybrid Pro-social Behaviour Programme: Coping Power and Strengthening Families Program

• Aimed at 9 and 10 year olds

• Universally applied in schools in Tallaght

• Parental Components

• Designed to enhance children’s pro-social development, reduce anti-social behaviour, improve parenting skills and parent/child interactions


Common Methodology• Individually randomised trials

• Qualitative Process Evaluation

• Children recruited on basis of teacher referrals

• 3 year rolling cohort design

• Over 600 children allocated to either intervention or to control: ‘business as usual’

• Child, parent and teacher measures

• Outcomes measured according to pre-validated measures where possible.


Summary Outcome Results – Doodle Den

• Primary Outcomes:– Statistically significant effects on both direct child and teacher assessed measures of literacy

(d=0.17-0.3)

• Secondary Outcomes:– Statistically significant effect on children’s behaviour in regular school class (d=0.18)

– No impact on attendance rates at school

• Exploratory Analysis– No significant differences in terms of literacy

– Boys behaviour improved relative to control group

– Greater attendance at the programme led to more positive outcomes


Summary Outcome Results – Mate Tricks

• Primary Outcomes:– No statistically significant effects on pro-social behaviour

– Two significant negative effects on anti-social behaviour (d=0.18; 0.20)

• Secondary Outcomes:– No statistically significant positive effects on wide range of secondary measures

– Some significant negative effects in relation to child reported parenting practices

• Exploratory Analysis– Child programme attendance not related to outcomes

– Parental engagement with the programme significant predictor of a range of positive outcomes


Conextualising Results – Doodle Den

• First randomised trial of an after-school program in UK or Ireland focused on academic outcomes

• Mixed results of previous trials of the effectiveness on academic outcomes

– e.g. 21st Century schools U.S. national evaluation

• Reviews of evidence highlight complexity of the field

• Recent promising practice guides have concluded a need for much more research on what works in after-school settings

• Potential features driving positive outcomes: – Age of children; targeted at those most in need, well structured clearly defined curriculum,

qualified staff with focus on professional development and fun engaging activities


Conextualising Results – Mate Tricks

• Hybrid programme with some evidence of effectiveness behind the two programmes

• No evidence of effectiveness of Mate Tricks in improving behavioural outcomes and some negative effects apparent

• Some evidence of in-school programmes focusing on social and emotional learning having promising effects e.g. Incredible years

• However, previous reviews have also highlighted null or adverse effects in after-school contexts

• The current study adds to existing evidence that behavioural programmes have the potential to cause adverse effects as well as benefits

• r


Some Lessons Learned: Methodological Issues

Despite the critics, it is possible to implement RCTs in real world community settings.

Making key stakeholders aware of key principle’s of RCT evaluation designs A willingness to be flexible, without undermining the core of the study and randomisation process

Developing and maintaining good relationships

Keeping attrition to a minimum without overburdening participants

Importance of measurement and agreeing analysis plan in advance

RCT’s good at assessing intended outcomes but good process data should not be neglected


Ethical Issues and RCT’sEquipose

Randomisation perceived as a fair way of allocation

Ethical Issues may arise at any stage

Potential for burden

Dissemination and not spoiling the field, especially when faced with null or negative findings


EVIDENCE MATTERS!!!We intervene in children’s lives on a regular basis!

We may also do damage when we intend to do good!

Interventions often driven by fashion rather than sound evidence!

Do we owe it to children to try and ensure that the policies they are subjected to are effective in improving their life chances?

Both reports available on CEE website: www.qub.ac.uk/cee

transforming education through evidence

Documents

biomedical sciencesthe

school year3 days

regular school class

significant effects

antisocial behaviour

prosocial development

teacher measuresoutcomes

secondary outcomes