bowen & neill (2013) adventure therapy meta-analysis presentation
TRANSCRIPT
Daniel J. Bowen
Registered Psychologist
PhD Candidatewww.danielbowen.com.au/
Dr James T. Neill
Associate Professorwww.wilderdom.com/
A Meta-Analysis of Adventure Therapy Outcomes and Moderators
University of Canberra
AustraliaBowen, D. J., & Neill, J. T. (2013). A meta-analysis of
adventure therapy outcomes and moderators.
The Open Psychology Journal, 6, 28-53.
doi: 10.2174/1874350120130802001
July, 2016
Meta-Analysis
A meta-analysis is:
“a set of statistical methods
for combining quantitative results
from multiple studies
to produce an overall summary
of empirical knowledge
on a given topic”
(Littell, Corcoran & Pillai, 2008, p. 1-2)
Effect Sizes (ES)
Effect size (ES) (Ellis, 2010):
= Magnitude of the treatment effect
Calculated for each study
Combined to compute a summary effect
Effect Size Statistic Used: Hedges’ g
= Difference between the means of 2 groups ÷ population standard
deviation: (M1-M2)/Sdpooled
Adjusts for studies with small sample sizes
Interpreting: 0.20 (small), 0.50 (medium), 0.80 (large)
Effect Size Confidence Intervals (CIs)
Confidence Interval (CI) (Ellis, 2010):
A range of values that describes the uncertainty surrounding an estimate
(e.g., of an effect size (ES))
Indicates a range of values that's likely to encompass the true value
Is also itself an estimate
– The two numbers that make up the lower and upper ends of the CI are called the
lower and upper confidence limits. The lower and upper figures provides an
indication of the expected or possible range surrounding the true value.
95% CIs (used in this meta-analysis) contain the true value 95% of the time
and fail to contain the true value the other 5% of the time.
If the CI excludes the null value of zero, then the mean effect size is
considered to be statistically significant
Significant ES Non-Significant ES
ES = 0.40 (CI.025= +0.20; CI.975= +0.80) ES = 0.40 (CI.025= -0.40; CI.975= +0.40)
Justification & Aim
In the 21st century, it is necessary for rigorous research to
document the effect of adventure therapy programs, and
analyse the contributing factors.
• For: Boards, treatment facilities, funding bodies, policy makers
and consumers.
There are currently no published meta-analyses directly
concerned with adventure therapy, across the life span.
Broadly, this study aimed to:
Consolidate and provide up-to-date information regarding the
therapeutic uses and treatment effectiveness of adventure therapy
Provide valuable insight regarding the utility and therapeutic
outcomes of AT programs
Research QuestionsMeta-Analysis of Adventure Therapy Program Effects
1. What are the effects of adventure therapy?- What are the overall psychotherapeutic and psychosocial effects
of adventure therapy?
- What are effects on specific types of outcomes?
- How do the effects of adventure therapy compare to adventure
education?
- How do the effects of adventure therapy compare to other forms
of therapy?
2. What are the major correlates of adventure therapy
program outcomes? For example:- Outcomes (e.g., clinical, behavioural, family, social, academic)
- Study (e.g., publication year/type; size, quality of research)
- Participants (e.g., age, race, gender)
- Program (e.g., funding, delivery, group structure, model, length)
Selection Criteria
Included studies must:
Primarily use adventure activities specifically for
psychological and/or behavioural therapeutic purposes
Report at least pre- and post-psychological and/or
behavioural outcomes
Provide sufficient statistical information to allow
calculation of standardised mean effect sizes
– E.g., means, standard deviations, number of participants
Be reported in 1960 or later, and in English
Adapted from: George (2011); Wilson & Lipsey (2000)
Search Strategy & Data Extraction
Completed a comprehensive search:
– Electronic data bases
– Related journals, internet sites, meta-analyses
– Listserv and emails to experts in the field
– Scanned bibliographies and reference lists
Identified and obtained studies were included or
excluded based on the selection criteria.
Data was extracted from included studies via a
Coding Manual (Adapted from George (2011) and Wilson & Lipsey (2000))
Data AnalysisFollowed the Method of Borenstein et al. (2009)
Completed using Comprehensive Meta-Analysis - Version 2
The random-effects model was used
A single mean effect size from each sample was calculated
(where available) for each:
– Treatment group: Adventure; Alternative; No
– Outcome category:
Academic, Behaviour, Clinical, Family Development, Morality/Spirituality,
Physical, Self-Concept, and Social Development
– Time Comparison: Base-Pre; Pre-Post; Post-Follow-Up
– Overall effect
Additional comparisons were conducted to explore the
existence of potential moderators
Results Based On
2,908 effect sizes
From 206 unique samples located within 197 studies
Published between 1967 and 2012
Treatment Group No. & %
Adventure Therapy 2,275; 78.2%
Alternative Therapy 335; 11.5%
No Therapy 298; 10.3%
Time Comparison No. & %
Base-Pre 55; 1.9%
Pre-Post 2,274; 78.2%
Post-Follow-Up 579; 19.9%
Europe NZ Asia Canada Australia USA
2
(1.0%)
3
(1.5%)
7
(3.4%)
7
(3.4%)
26
(12.6%)
161
(78.2%)
Results
Based On:
Participants: ~ 17,728 (M = 86.1; SD = 148.3 per study)
Gender: male (62.5%) and female (37.5%)
Age: between 9 and 64 years (M = 16.9; SD = 7.0)
Program location: USA (78%), Australia (13%), Canada (3%),
Asia (3%), NZ (2%), & Europe (1%)
Program Model: Ropes/Challenge/Adventure-Based (42%),
Expedition (27%), Mixed (21%), Base Camp (5%), Residential
(4%), & Outpatient (1%).
Program length: 1-534 days (M = 63.5 days; SD = 148.0)
Mean time b/w Base-Pre: 20.8 days (SD = 15.3)
Mean time b/w Post-Follow-up: 181.4 days (SD = 276.0)
Overall Effect Sizes for Treatment Group by Time Comparison
Note: Caution should be used when interpreting some of the results due to high heterogeneity.
Interpreting g: 0.20 (small), 0.50 (medium), 0.80 (large)
0.09
0.56 0.59
0.14
-0.03
0.08
-0.08-0.20
0.00
0.20
0.40
0.60
0.80
Base-Pre Base-Post Base-FU
Adventure Tx Alternative Tx No Tx
Pre-Post Adventure Therapy Treatment Effect by Outcome Category
Hedges g ↓ 95% CI ↑ 95% CI
0.53
0.59
0.490.49
0.50 0.51 0.50
0.55
0.35
0.470.50
0.43 0.42 0.41 0.410.36
0.32
0.17
0.410.42
0.36 0.340.33
0.31
0.23
0.10
-0.01-0.05
0.05
0.15
0.25
0.35
0.45
0.55
Overall Clinical Self-Concept
SocialDevelopment
Academic Behaviour FamilyDevelopment
Physical Morality/Spirituality
Post-Follow-up Adventure Therapy Treatment Effect by Outcome Category
Hedges g ↓ 95% CI ↑ 95% CI
0.09
0.70
0.31
1.00
0.15 0.110.06 0.08 0.050.03
0.230.21 0.11
0.05 0.01-0.03 -0.05 -0.06-0.04
-0.24
0.12
-0.77
-0.07 -0.08-0.12 -0.17 -0.17
-0.80
-0.60
-0.40
-0.20
0.00
0.20
0.40
0.60
0.80
1.00
Overall Clinical Self-Concept
SocialDevelopment
Academic Behaviour FamilyDevelopment
Physical Morality/Spirituality
Publication Bias Results
Publication Bias was Assessed Using the Trim & Fill Method
Suggested 18 studies were missing
With:
g = .52
Without:
g = .47
Moderator Results
Conducted for Pre-Post Adventure Therapy Treatment
In most instances, there are no clear or notable trends
In general, higher effects were found for:
– Samples which were published or had a lower quality rating
– Programs whose use of adventure was adjunctive, or had an
open group structure or private placement
– Participants who were older or female
∴ On the whole, these variables do not greatly moderate
the effectiveness of adventure therapy programs
∴ Provide further support for the overall moderate
effectiveness of adventure therapy.
Meta-Regression Results
Meta-Regression:
To investigate whether sample-, program- and
participant-level characteristics explain the variation
(heterogeneity) of results among studies.
Results:
Age was the only significant predictor
Accounted for 6.8% of variance
Stronger outcomes for older participants
Sample: 5.6% Program: 11.5% Participant 27.5%
Pre-Post Adventure Therapy Treatment Effect by Age
Hedges g ↓ 95% CI ↑ 95% CI
0.53
0.44 0.44
0.59
0.921.00
0.47
0.24
0.37
0.50
0.66
0.66
0.41
0.04
0.30
0.41 0.40
0.31
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
All Ages ≤9 Years Old 10-14 YearsOld
15-17 YearsOld
18+ Years Old Mixed(e.g., Families)
Comparison of Different Types of Therapy
Note: 1) As there are several noteworthy differences between adventure therapy and more traditional forms of psychotherapy, comparison of meta-analytic
findings is problematic and should be interpreted with caution, and 2) as confidence intervals overlap, there are no significant differences between treatment types.
Benchmarking(Neill, 2003)
Results support the use of findings from this study for
benchmarking program effectiveness.
Obtained by comparing a particular program’s effect
sizes with appropriate meta-analytic benchmarks.
Provides a systematic means to categorise and evaluate
outcomes.
Provides valuable diagnostic information about a
program’s relative efficacy
Aged-based benchmarks are available from
http://www.danielbowen.com.au/meta-analysis
Limitations
Limitations of this Study & Meta-analytic Analysis in General:
Availability of studies
Heterogeneity
Generalisability
Type of data provided by empirical studies
– Including the provision of sufficient statistical information to
calculate effect sizes
Methodological quality of studies
– Including the use of psychometrically validated assessment tools
Recommendations for Future Research
Report matched samples, and enumerate drop outs.
Provide the necessary statistics to allow calculation of effect sizes.
Provide sample, participant and program demographics.
Capture data about a larger range of outcomes at multiple time
points.
Consult such resources as the Methodological Quality Rating
Scale or Rubric for Evidence-Based Research on Adventure
Programs (Gass et al., 2012) when designing a research project.
Use psychometrically validated assessment tools.
Rigorously develop purpose-built, multidimensional assessment
tools using the best available psychometric techniques.
Conclusion
Most substantial and robust meta-analysis of adventure
therapy program outcomes to date.
Confirms that overall adventure therapy is moderately
effective in facilitating positive short-term change.
On average, effects appear to be maintained in the long-
term.
Heterogeneity of outcomes indicates considerable
variability, although little of variance appears to be
explained by the measured moderators.
Key References
Borenstein, H., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. (2005). Comprehensive
meta-analysis (Version 2). Englewood, NJ: Biostat.
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to
meta-analysis. Chichester, UK: John Wiley & Sons.
Bowen, D. J., & Neill, J. T. (2013). A meta-analysis of adventure therapy outcomes and
moderators. The Open Psychology Journal, 6, 28-53. doi:
10.2174/1874350120130802001
Gass, M. A., Gillis, H. L., & Russell, K. C. (2012). Adventure therapy: Theory, research,
and practice. New York: Routledge.
Hattie, J. M., Marsh, H. W., Neill, J. T., & Richards, G. E. (1997). Adventure education
and Outward Bound: Out-of-class experiences that make a lasting difference. Review
of Educational Research, 67(1), 43-87. doi: 10.3102/00346543067001043
Littell, J. H., Corcoran, J., & Pillai, V. K. (2008). Systematic reviews and meta-analysis.
New York: Oxford University Press.
Neill, J. T. (2003). Reviewing and benchmarking adventure therapy outcomes:
Applications of meta-analysis. Journal of Experiential Education, 25(3), 316-321.