11

Educational Psychology in Scotland Educational Psychology in Scotland Annual Conference Annual Conference

Herriot Watt UniversityHerriot Watt UniversitySeptember 14September 14thth 2012 2012

The Preschool PATHS Curriculum:The Preschool PATHS Curriculum:

Using Implementation Science to Increase Using Implementation Science to Increase EffectivenessEffectiveness

Dr Barbara Kelly, Carole Edgerton, Dr Barbara Kelly, Carole Edgerton, Elaine Robertson, Diann NeilElaine Robertson, Diann Neil

Glasgow South Psychological Glasgow South Psychological ServicesServices

Outline of PresentationOutline of Presentation

• What is PATHS?What is PATHS?

• Preschool PATHS project in Glasgow SouthPreschool PATHS project in Glasgow South

• Focus on Implementation ScienceFocus on Implementation Science

• Interim Audit of IMPLEMENTATIONInterim Audit of IMPLEMENTATION

• Glasgow PATHS Network Glasgow PATHS Network Building Sustainable Programmes

• Fuller discussion and references for all the issues raised can be found in Barbara Kelly and Danny Perkins 2012, Cambridge Handbook of Implementation Science for Psychology in Education

22

33

PATHS Curriculum in PATHS Curriculum in GlasgowGlasgow• Fairer Scotland Preschool PATHS project one of few Fairer Scotland Preschool PATHS project one of few

externally funded in Glasgow`s education contextexternally funded in Glasgow`s education context• PATHS primary curriculum is becoming well known PATHS primary curriculum is becoming well known

throughout Scotland and has a developing evidence throughout Scotland and has a developing evidence base locally helping to fulfil Glasgow`s vision for base locally helping to fulfil Glasgow`s vision for children's well being and welfarechildren's well being and welfare

• Preschool PATHS is a fairly recent development in Preschool PATHS is a fairly recent development in Glasgow and is currently being embedded in a Glasgow and is currently being embedded in a number of preschool establishments number of preschool establishments

• Preschool project focuses on Implementation Preschool project focuses on Implementation Science with a view to increasing sustainability and Science with a view to increasing sustainability and maximising effectivenessmaximising effectiveness

44

Why choose PATHS and Why choose PATHS and Preschool PATHS?Preschool PATHS?• Recognised as one of only eleven `blueprint` programmes Recognised as one of only eleven `blueprint` programmes

recommended for reducing adolescent crime, aggression and recommended for reducing adolescent crime, aggression and substance abuse in the US (Mihalic et al 2001)substance abuse in the US (Mihalic et al 2001)

• Significant positive outcomes for ASN pupils and in mental health Significant positive outcomes for ASN pupils and in mental health contexts ( Greenberg et al, 2000)contexts ( Greenberg et al, 2000)

• Study in England demonstrated greater cooperation, empathy and Study in England demonstrated greater cooperation, empathy and self control (Curtis and Norgate, 2007)self control (Curtis and Norgate, 2007)

• In Scotland enhanced pupils` ability to manage difficult emotion In Scotland enhanced pupils` ability to manage difficult emotion and positive impact on behaviour ( Kelly et al, 2004)and positive impact on behaviour ( Kelly et al, 2004)

• Key paper for Preschool PATHS effectiveness Domitrovich et al 2009Key paper for Preschool PATHS effectiveness Domitrovich et al 2009

Reflects government priorities for developing Reflects government priorities for developing children`s mental health and well beingchildren`s mental health and well being

55

Why choose PATHS?Why choose PATHS?

• Evidence of impact in primary and preschool Evidence of impact in primary and preschool PATHS is building globallyPATHS is building globally

• Underlying dynamic, theoretical framework Underlying dynamic, theoretical framework proving to be proving to be more and more relevantmore and more relevant to our to our understanding of child development and understanding of child development and learninglearning

• The central concepts of The central concepts of integration of integration of functioningfunctioning and the importance of and the importance of self- self- regulationregulation have anticipated research findings have anticipated research findings and are key emerging theoretical models in and are key emerging theoretical models in developmental psychology developmental psychology

66

PATHS ImpactsPATHS Impacts

• Offers a range of strategies to recognise Offers a range of strategies to recognise manage and regulate feelings and develop manage and regulate feelings and develop alternatives to hostility and violencealternatives to hostility and violence

• Impacts on attainments –literacy and languageImpacts on attainments –literacy and language• Linked closely to cognitive skills and problem Linked closely to cognitive skills and problem

solving: emotional understanding underpins solving: emotional understanding underpins effective thinkingeffective thinking

But emotional education may have wider effects: But emotional education may have wider effects: literature from theory of mind research and literature from theory of mind research and developmental psychology suggests that emotion developmental psychology suggests that emotion content of conversations between parents and children content of conversations between parents and children creates feeling, thinking and behavioural templates creates feeling, thinking and behavioural templates

PATHS in ACTION 1

PATHS in ACTION 2

PATHS in ACTION 3

1313

Focus on Implementation Focus on Implementation ScienceScience

The way a programme is implemented The way a programme is implemented is more important than the programme is more important than the programme itselfitself•Implementation is an emerging Science- Journal Implementation is an emerging Science- Journal of Implementation Science appeared in 2006of Implementation Science appeared in 2006•Barbara Kelly and Danny Perkins Barbara Kelly and Danny Perkins (2012),Cambridge Handbook of Implementation (2012),Cambridge Handbook of Implementation Science for Psychology in Education, Cambridge Science for Psychology in Education, Cambridge University PressUniversity Press•Offers conceptual and applied frameworks and Offers conceptual and applied frameworks and strategies for effective implementationstrategies for effective implementation

1414

Basic Implementation Strategies Basic Implementation Strategies (Blasé et al 2012) Who, why and (Blasé et al 2012) Who, why and what in building effectivenesswhat in building effectiveness1.1. Recruitment and selectionRecruitment and selection: choose those who are : choose those who are

interested in the programme and keen to be involved.interested in the programme and keen to be involved.2.2. Training:Training: provide in depth pre and in service training on both provide in depth pre and in service training on both

theory and practice issues linked to the programme theory and practice issues linked to the programme 3.3. On going Consultation and CoachingOn going Consultation and Coaching: follow through initial : follow through initial

training with coaching support in context for practitioners.training with coaching support in context for practitioners.4.4. Monitor and evaluate Staff performanceMonitor and evaluate Staff performance: monitor how : monitor how

staff use skills to target input at all stagesstaff use skills to target input at all stages5.5. Inform decision making and policy reviewInform decision making and policy review: provide data to : provide data to

support on-going local investment decisionssupport on-going local investment decisions6.6. Facilitative administrative supportFacilitative administrative support: have clear leadership : have clear leadership

and support to implement effectively and sustain programmes and support to implement effectively and sustain programmes effectivelyeffectively

7.7. System SupportSystem Support: Ensure the availability of financial, : Ensure the availability of financial, organisational and other resources- don’t run on a shoe stringorganisational and other resources- don’t run on a shoe string

1515

Design of the project Design of the project Every new implementation of any programme Every new implementation of any programme presents a new ecology and different strengths and presents a new ecology and different strengths and weaknessesweaknesses

• Two year implementation process Two year implementation process • This reports on first year auditThis reports on first year audit• Used Audit Methodology - support and Used Audit Methodology - support and

check the progress made in implementing check the progress made in implementing evidence based programmes and allows evidence based programmes and allows feedback and changes to improve feedback and changes to improve effectivenesseffectiveness

• Monitors successes and obstacles along Monitors successes and obstacles along the waythe way

• Poor implementation results in low impactPoor implementation results in low impact

1616

So what have we actually done?So what have we actually done?

• Following the Implementation Drivers identifies by Blase et al Following the Implementation Drivers identifies by Blase et al 2012 we focused on key areas to support implementation:2012 we focused on key areas to support implementation:

First -Set up a steering group involving Principal Psychologist, First -Set up a steering group involving Principal Psychologist, Senior EP, EPs, Quality Improvement Officer, Early Years Co-Senior EP, EPs, Quality Improvement Officer, Early Years Co-ordinator to guide the implementation processesordinator to guide the implementation processes

Recruitment and initial training processes for staffRecruitment and initial training processes for staff

• Presentation to volunteer nursery head teachersPresentation to volunteer nursery head teachers

• Completion & return of the Readiness ChecklistCompletion & return of the Readiness Checklist

• Selection of nurseriesSelection of nurseries

• 4 nurseries with 10 children in each as the in-depth research 4 nurseries with 10 children in each as the in-depth research group group

Assessing Readiness Aarons Assessing Readiness Aarons et al 2012et al 2012• Readiness for evidence based practice is a Readiness for evidence based practice is a

concept developed by Aarons concept developed by Aarons

• Arose from medical context where Arose from medical context where practitioners were unable or unwilling to practitioners were unable or unwilling to implement evidence based programmes implement evidence based programmes successfullysuccessfully

• Reasons are linked to processes outlined Reasons are linked to processes outlined earlier: values , attitudes, understanding, earlier: values , attitudes, understanding, resources, leadership………and `diamond resources, leadership………and `diamond hard resistance`hard resistance`

1717

PATHS readiness checklistPATHS readiness checklist

• Asked volunteers to consider a range of Asked volunteers to consider a range of issues which they would have to address issues which they would have to address to implement PATHS effectivelyto implement PATHS effectively

• This was used to select candidates for This was used to select candidates for the study the study

• Not an infallible method as reality always Not an infallible method as reality always proves more challenging but alerts proves more challenging but alerts practitioners to implementation demandspractitioners to implementation demands

1818

1919

Extensive data collection Extensive data collection throughout and at end of year throughout and at end of year oneoneImpact on childrenImpact on children• Kusche Emotional Inventory (Domitrovich et al)Kusche Emotional Inventory (Domitrovich et al)• The British Picture Vocabulary Scale 3The British Picture Vocabulary Scale 3• Demographic QuestionnaireDemographic Questionnaire• Preschool Behaviour ChecklistPreschool Behaviour Checklist• Number of children involved was based on the statistical analysis Number of children involved was based on the statistical analysis

of the Domitrovich study – 18 children required to demonstrate of the Domitrovich study – 18 children required to demonstrate effects.effects.

• Within Subjects Design Within Subjects Design to evaluate the impact of Pre-school to evaluate the impact of Pre-school PATHS This design makes the assumption that we know Pre-school PATHS This design makes the assumption that we know Pre-school PATHS works and looks at its effects on individual children in PATHS works and looks at its effects on individual children in different early years contexts. different early years contexts.

2020

Implementation, training Implementation, training and support activitiesand support activities• Major innovation in Glasgow South -Major innovation in Glasgow South -Coaching Coaching

and Modelling by Peripatetic Preschool Teamand Modelling by Peripatetic Preschool Team• Inset to all nursery staff on Pre-school PATHS & Inset to all nursery staff on Pre-school PATHS &

Implementation IssuesImplementation Issues• Signed agreement to follow implementation Signed agreement to follow implementation

guidelinesguidelines• Discussion with staff about frequency and quality Discussion with staff about frequency and quality

of lessons (30 required for significance)of lessons (30 required for significance)• Observation of some lessons for fidelity and Observation of some lessons for fidelity and

engagement lessonsengagement lessons• Interview with staff about impact after year oneInterview with staff about impact after year one

2121

Statistical Analysis after one Statistical Analysis after one yearyear• Repeated measures ANCOVA looked at scores in Repeated measures ANCOVA looked at scores in

Kushe Emotional Inventory and the British Picture Kushe Emotional Inventory and the British Picture Vocabulary Scale (BPVS). Vocabulary Scale (BPVS).

• The co-variants controlled were: gender, age, The co-variants controlled were: gender, age, number of lessons taught, English as an number of lessons taught, English as an Additional Language (EAL), ethnicity and Additional Language (EAL), ethnicity and Additional Support Needs (ASN). Additional Support Needs (ASN).

• Firstly the nursery results were looked at together Firstly the nursery results were looked at together as a whole population (n=30).as a whole population (n=30).

• Only thirty of the 40 target children had a full Only thirty of the 40 target children had a full data setdata set

2222

Emerging positive trendsEmerging positive trends

• The results of the ANCOVA for the The results of the ANCOVA for the Kushe Emotional InventoryKushe Emotional Inventory showed showed no no significantsignificant overall effect of time at overall effect of time at this stage; F(1,23)=0.881, p=0.358. this stage; F(1,23)=0.881, p=0.358.

• However, the mean percentage of However, the mean percentage of correct answers on both did increase correct answers on both did increase from time 1 (mean= 59.83%) to time from time 1 (mean= 59.83%) to time 2 (mean=63.33%). 2 (mean=63.33%).

2323

Checking by establishmentChecking by establishment

• T- Test T- Test significantsignificant on both the BPVS on both the BPVS and Kushe in two establishments but and Kushe in two establishments but low numbers of children hampers low numbers of children hampers confident interpretationconfident interpretation

• Backward trends began to emerge in Backward trends began to emerge in two establishments two establishments

2424

Interim and End of Year One OutcomesInterim and End of Year One Outcomes

Staff views via telephone interviews Staff views via telephone interviews • Overall feedback is very positive. Overall feedback is very positive. • All note positive changes in children's emotional and All note positive changes in children's emotional and

social development and behaviour. social development and behaviour. • All felt PATHS achieved its theoretical objectives very All felt PATHS achieved its theoretical objectives very

well.well.• Respondents emphasised key processes of children`s Respondents emphasised key processes of children`s

identification and empathy with puppet characters and identification and empathy with puppet characters and the value of scenarios played out by characters which the value of scenarios played out by characters which generalised to real situations with teacher promptsgeneralised to real situations with teacher prompts

• All will continue into year two and report developing All will continue into year two and report developing confidence in delivery and management of the lessonsconfidence in delivery and management of the lessons

Little data to date on the early stages of Little data to date on the early stages of implementation-the installation and initial implementation-the installation and initial implementation phases (Schaffer, Nesselrodt and implementation phases (Schaffer, Nesselrodt and Stringfield (1997)Stringfield (1997)

• Many programmes fail during the installation Many programmes fail during the installation phase because of lack of investment of time phase because of lack of investment of time and resourcesand resources

• Leadership role is crucial in maintaining a focus Leadership role is crucial in maintaining a focus on quality implementation and challenging on quality implementation and challenging current values, beliefs and paradigms which do current values, beliefs and paradigms which do not fit the new way of workingnot fit the new way of working

• No one has a crystal ball and this phase is characterised No one has a crystal ball and this phase is characterised by rapid problem solving cycles which respond to issues by rapid problem solving cycles which respond to issues emerging in the audit processemerging in the audit process

2525

Why were results not reaching Why were results not reaching significance? Audit of early significance? Audit of early implementation stagesimplementation stages

Training-Training-not all received follow up trainingnot all received follow up training

Number of lessons- Number of lessons- no establishment no establishment carried out the planned thirty lessonscarried out the planned thirty lessons

Coaching-Coaching- varied in style, frequency and varied in style, frequency and experience: some staff began doing lessons experience: some staff began doing lessons themselves when the agreement had been themselves when the agreement had been on modelling in the early stages, some did on modelling in the early stages, some did not find the coach helpful.not find the coach helpful.

2626

Implementation and child Implementation and child resultsresults• These are reassuringly in a strong positive These are reassuringly in a strong positive

direction in two establishmentsdirection in two establishments

• In the other two establishments problems In the other two establishments problems arose probably impacting on child resultsarose probably impacting on child results

Staff continuity, knowledge and experience of Staff continuity, knowledge and experience of coaches, knowledge and training of staff, coaches, knowledge and training of staff, frequency of lessons, resource issues and frequency of lessons, resource issues and confidence, fidelity of lessons taught.confidence, fidelity of lessons taught.

2727

2828

Staff reactions to Staff reactions to Implementation IssuesImplementation Issues• Majority nurseries felt that the in-service Majority nurseries felt that the in-service

presentation and taster sessions fired imaginationpresentation and taster sessions fired imagination• Coaches too slow to get off the ground re lessonsCoaches too slow to get off the ground re lessons• Some difficulties with language barriers – high Some difficulties with language barriers – high

numbers of EAL parents and children in one numbers of EAL parents and children in one nurserynursery

• Two nurseries reported feeling unsupported initially Two nurseries reported feeling unsupported initially and that the coach had poor skillsand that the coach had poor skills

• Resource and time issues re organisation and Resource and time issues re organisation and photocopyingphotocopying

• Difficulties across job share staff and Difficulties across job share staff and communication with themcommunication with them

• Extension activities were too difficult.Extension activities were too difficult.

2929

Reaction of parentsReaction of parents• Nurseries reported that parents seemed to Nurseries reported that parents seemed to

be interested and enthusiastic but they be interested and enthusiastic but they hadn’t set out to involve themhadn’t set out to involve them

• One set up a PATHS display and this had One set up a PATHS display and this had stimulated a lot of conversationstimulated a lot of conversation

• One reported that parents had come and One reported that parents had come and asked to see the materialsasked to see the materials

• Staff feel that parent involvement would Staff feel that parent involvement would be helpful in the implementation phasesbe helpful in the implementation phases

3030

What’s gone well?What’s gone well?• Most reported coach had been excellent Most reported coach had been excellent

and built confidence in staff to deliver on and built confidence in staff to deliver on their owntheir own

• Programme is really clear and very Programme is really clear and very comprehensivecomprehensive

• Materials are excellent and children are Materials are excellent and children are really enjoying the programmereally enjoying the programme

• Two nurseries report evidence of children Two nurseries report evidence of children discussing PATHS at home with parentsdiscussing PATHS at home with parents

• Three feel impact is clear alreadyThree feel impact is clear already

Next phaseNext phase

• Each establishment to complete 30 lessons before child Each establishment to complete 30 lessons before child assessmentassessment

• Iron out problems via close collaboration between nursery Iron out problems via close collaboration between nursery EP and staffEP and staff

• Carry out a final audit after two years or thirty lessons and Carry out a final audit after two years or thirty lessons and when staff are seen to deliver lessons with fidelity and when staff are seen to deliver lessons with fidelity and confidenceconfidence

• Established PATHS Network to foster good implementation Established PATHS Network to foster good implementation of PATHS and other programmes, promote self evaluation of PATHS and other programmes, promote self evaluation and to develop links to global PATHS networksand to develop links to global PATHS networks

3131

Implementation Science: Implementation Science: The Practitioner is the The Practitioner is the interventionintervention• Offers a sound evidence base and guidelines for Offers a sound evidence base and guidelines for

building accountable evidence based practice building accountable evidence based practice across all aspects and levels of Psychological across all aspects and levels of Psychological Service DeliveryService Delivery

• Success for children ultimately depends on staff Success for children ultimately depends on staff readiness to deliver high quality evidence based readiness to deliver high quality evidence based teaching, care and programmes with enthusiasm, teaching, care and programmes with enthusiasm, depth of understanding of theory and skill.depth of understanding of theory and skill.

• Supporting those processes is a major part of our Supporting those processes is a major part of our work but is demanding and intricate. work but is demanding and intricate.

3232