the campbell collaboration some challenges in reviewing the effectiveness of behavioural group-based...

The Campbell Collaboration www.campbellcollaboration.org

Some challenges in reviewing the effectiveness of behavioural group-based parenting programmes for

childhood conduct problems

Mairead Furlong 1, Sinead McGilloway 1, Tracey Bywater 2, Judy Hutchings 3, Susan Smith 4, Michael Donnelly 5

Institutional affiliations: 1 National University of Ireland Maynooth; 2 University of York, England; 3 Bangor University, Wales; 4 Royal College of Surgeons in Ireland; 5 Queens University Belfast, N.Ireland

Cochrane Colloquium 2012


Overview

• Background – why the review was conducted

• Brief overview of inclusion criteria and results of review

• Some methodological challenges


Background

• Conduct problems in young children are common – 5% to 10% in UK, US and Europe

• Rates in socially disadvantaged areas -> 20% to 35%

• Negative long-term psychological, social and economic consequences (e.g. Farrington, 2007)

• Evidence that group-based parenting programmes, based on behavioural and social learning principles may reduce conduct problems (e.g. Webster-Stratton, 1988; Sanders, 2000)


Background – cont’d

• Previous systematic reviews have been problematic with regard to:

- lack of statistical meta-analyses and defined against established criteria as ‘effective’ (eg. Brestan & Eyberg,1998; Mihalic et al.2002) - failure to report heterogeneity and confidence intervals

- no intention-to-treat or sensitivity analyses undertaken - results of group-based behavioural programmes combined with other

types of parenting programmes (e.g. individual programmes; different theory base; those with adjunctive treatments)

• Thus, the evidence for behavioural group-based parenting programmes remains unclear


Objectives and inclusion criteria

To examine the effectiveness and cost-effectiveness of behavioural and cognitive-behavioural group-based parenting programmes for children (aged 3-12 year) with early-onset conduct problems in improving child conduct problems, parental mental health and parenting skills

• Studies were included if: (1) Randomised or quasi-randomised controlled trial; (2) Based on parent interventions only (i.e. did not involve children); (3) Incorporated an intervention group versus a waiting list, no treatment or

standard treatment control group. Excluded parent programme as control


Inclusion/exclusion criteria

Included studies that: • Used at least one standardised instrument to measure clinically significant child conduct problems.• Reported on conduct problems co-morbid with ADD and ADHD if they

reported outcomes for conduct problems separately from ADD and ADHD outcomes

• Excluded studies that involved children with comorbid physical and intellectual impairments (e.g. ASD, Down Syndrome, tic disorders, significant language delay and learning problems)

• Only included costs or cost-effectiveness studies conducted alongside, or subsequent to, RCTs that met our eligibility criteria


13 effectiveness studies met inclusion criteria – 10 full RCTs, 3 quasi-RCTs; two cost-effectiveness studies included (Incredible Years studies)

1078 participants (646 intervention, 432 control)

5 studies conducted in USA, 7 in Europe and 1 in Australia

In 6 studies, children had a diagnosis of Conduct Disorder; in 7 studies they scored above clinical cut-off on a validated measure of childhood conduct problems

9 studies were Incredible Years; 1 Triple P; 1 Barkley’s Parent Training Programme; 1 Parent Management Training and 1 Comet Parent Management Training


Results cont’d

• All measures were short-term outcomes

• Moderate statistically significant improvements in child conduct problems, whether assessed by parents (SMD -0.53; 95% CI -0.72 to -0.34) or independently (SMD -0.44; 95% CI -0.77 to -0.11)

• Sub-moderate improvements in parental mental health (SMD -0.36; 95% CI -0.52 to -0.20) and moderate improvements in positive parenting skills


• Results maintained across different settings (research and service), levels of disadvantage, diagnostic categories, and within services demonstrating high implementation fidelity

• Results remained robust to all but one of the sensitivity analyses

• Insufficient evidence to test for child internalising problems and cognitive/educational performance.

• The parent-training cost approximately $2500 (£1712/€2217) per family to bring the average child with clinically significant conduct problems into the non-clinical range.


Challenge 1

Were we right to exclude studies that did not provide evidence of children having clinical levels of conduct problems?

No:Exclusion of preventative studies; Exclusion of studies which indicated the inclusion of children with conduct problems but provided no evidence

Yes: Other reviews had combined preventative and clinical studies -> but results for programmes were unclear. We had a more focussed question


Challenge 2

Were we right to exclude children with comorbid physical and intellectual impairments?

No – ‘real world’ research suggests conduct problems often comorbid with learning difficulties

Yes - none of the excluded studies had minor comorbid learning difficulties but rather Downs Syndrome, Autism etc. Condition under investigation changes if those with severe impairments are included

Already too many sub-groups – precluded any subgroup analysis

based on children with mild learning/physical problems.


Challenge 3

Should we have included all children with ADHD as it’s often comorbid with conduct problems?

•Evidence on both sides to say that CP and ADHD overlap - but also different aetiology and treatment…

•Compromised => included studies with diagnosed ADHD if comorbid with conduct problems and if they reported separate outcomes for conduct problems and attention/hyperactivity

•Five included studies reported a low level of comorbidity with hyperactive symptoms; excluded in Zwi review as focus on CP


Challenge 4

How do you include all measures of an outcome within a study

into the review? •Typical advice in handbook…..

Parent Training Control Std. mean difference

Study or subgroup Mean Sd Total Mean Sd Total Weight IV, Random, 95% CI

Gardner 2006 66.7 29.9 34 73.5 34.7 26 22.30% -0.55 [-1.07, -0.03]

Hutchings 2007a 122.3 35.1 104 144 33 49 18.00% -0.63 [-0.97, -0.28]

Kling 2010 34.9 25.6 58 139.8 45.9 40 15.80% -0.77 [-1.19, -0.35]

Martin 2003 99.88 22.39 16 126.09 28.11 11 17.70% -1.02 [-1.85, -0.20]

McGilloway 2009 121.3 40.7 103 144.9 33.2 46 17.70% -0.61 [-0.96, -0.25]

Webster-Stratton 1984

106.51 20.33 13 138.91 20.33 11 7.70% -1.54 [-2.47, -0.61]


What would you do?

• Some other Cochrane reviews - included only the most commonly used measure (e.g. the Eyberg Child Behaviour Inventory)…

• We were wrong - focused only on means and SDs for continuous outcomes!

• Calculate mean effect size and SE for each individual outcome within each study; enter using generic inverse variance method (using SMDs)

Parent Control Std. mean difference

Study or subgroup Std. mean difference SE Total Total Weight IV, Random, 95% CIBarkley 2000 0.15 0.2206 39 42 15.30% 0.15 [-0.28, 0.58]Hutchings 2007a -0.43 0.1781 104 49 18.00% -0.43 [-0.78, -0.08]Kling 2010 -0.7 0.2128 58 40 15.80% -0.70 [-1.12, -0.28]McGilloway 2009 -0.59 0.1833 103 46 17.70% -0.59 [-0.95, -0.23]

Total (95% CI) 451 276 100.00% -0.50 [-0.76, -0.24]


Challenge 5

Reviewing costs studies = reviewing ‘in the dark’?• Completed both the ‘Drummond’ and ‘Evers’ checklists• Read all reviews conducted by Ian Shemilt and ‘Chapter 15’ of Handbook• A lack of standardised risk of bias table for costs studies

Key areas for improvement in costs studies in this area:• Need to adopt a ’complex intervention approach’ – wider range of costs/benefits• Reporting measures of variance for all parameters• Clearly delineating frequency of resource use from unit costs• Providing a more detailed description of the comparator and associated costs • Selecting outcome measures that can be compared with other studies


Conclusion: Other miscellaneous challenges and issues

• Inference challenges – lack of control group at long-term follow up

• Did not include qualitative research which can help to clarify adverse outcomes

• Our control group did not include other parenting programmes; did have treatment as usual…what should be in the control group?

Not a challenge• Did not expect to receive as much help from authors of included (and other)

studies so thank you!• Rewards of conducting a review in this area


With special thanks to…

• The Health Research Board in Ireland who funded the review (Cochrane Fellowship Scheme)

• The Cochrane Developmental, Psycho-social and Learning Problems group and most especially: Geraldine MacDonald, Laura MacDonald, Chris Champion, and Jane Dennis

• Anonymous peer reviewers for their helpful feedback

• The Campbell Collaboration


P.O. Box 7004 St. Olavs plass

0130 Oslo, Norway

E-mail: [email protected]

http://www.campbellcollaboration.org

the campbell collaboration some challenges in reviewing the effectiveness of behavioural group-based...

Documents