improving education through accountability and evaluation: lessons from around the world
DESCRIPTION
Building Evidence that Counts: Evaluating Career Pathways Interventions for Disconnected Youth and Adults in the U.S. David Fein Abt Associates Inc [email protected]. Improving Education through Accountability and Evaluation: Lessons from Around the World Rome, Italy 3 October, 2012. - PowerPoint PPT PresentationTRANSCRIPT
Improving Education through Accountability and Evaluation: Lessons from Around the WorldRome, Italy3 October, 2012
Building Evidence that Counts: Evaluating Career
Pathways Interventions for Disconnected Youth and
Adults in the U.S.
David FeinAbt Associates Inc
Acknowledgments
Work on Innovative Strategies for Increasing Self-Sufficiency (ISIS) evaluation sponsored by Administration for Children and Families, U.S. Department of Health and Human Services with major support from the Open Society Foundations, additional support from Joyce and Kresge foundations
Presentation represents the author’s views only Paper available at:
http://www.projectisis.org/wp-content/uploads/2012/05/ISIS-Career-Pathways-Framework-OPRE-2012-30_5-16-12.pdf
2
3
Introduction
Large population of adults and youth in the U.S. with low skills, little/no post-secondary education, and poor prospects for self-sufficiency
In the 1990s-2000s, national policies emphasized labor force attachment (versus education and training)
Evidence from social experiments influenced this emphasis– Consistent modest positive findings for employment-focused interventions;
disappointing findings for education and training– Some argued human capital investments should be limited to younger ages
Findings characterize traditional educational approaches not well-suited to second-chance populations
The Ground Is Shifting
Increasing evidence of limits to strictly employment-focused strategies, especially in a weak economy
Recession hits low-skilled jobs hardest and accelerates technological shifts widening skills premiums
Improved understanding of challenges facing individuals seeking post-2ndary, and deficiencies in second- (and first-) chance systems
Outpouring of promising innovations addressing these issues But these policies and systems remain small scale and highly fragmented… Enter Career Pathways – a body of ideas and practices for integrating
promising innovations promoted by a growing movement in the U.S. And, with work, a conceptual framework for building systematic, high-quality evidence on these ideas and practices
– It’s not about work vs. education, it’s about joining the two– I.e., “We’ve got the… stuff, we’ve just got to bring it together!” (Obama, on the
economy in general, 9-28-12)
4
Building Evidence on Career Pathways that Counts
Abt Associates, a global social policy research and technical assistance firm, is leading several national random assignment studies of promising career pathways interventions
The work has involved mapping out a conceptual framework and identifying priorities and opportunities for useful experiments
I will introduce this framework, show how it can help to organize a wide range of studies, and discuss some ways we are designing these projects so that their findings will count
5
Key Career Pathways Ideas
Address the wide range of skills and needs of second-chance populations
Create manageable, well-articulated training steps Provide credentials valued in high demand occupations/sectors Build effective partnerships
– E.g., public agencies (education, labor, human services); community colleges; community-based organizations; employers
– Vs. K-12, more diverse actors, complexity
Basic ideas also being applied in strengthening career pathways in secondary education
6
V. BA+ Programs Upper-Skilled Jobs V. BA+ Programs Upper-Skilled Jobs
IV. 1-2-Year Certificate to AA Programs Mid-Level Skilled Jobs
IV. 1-2-Year Certificate to AA Programs Mid-Level Skilled Jobs
III. Short-Term Certificate Programs Entry-Level Skilled Jobs
III. Short-Term Certificate Programs Entry-Level Skilled Jobs
II. Sectoral Bridge Programs Semi-Skilled Jobs
II. Sectoral Bridge Programs Semi-Skilled Jobs
I. Basic Bridge ProgramsI. Basic Bridge Programs
Pro
spec
ts f
or
go
od
-pay
ing
, st
ab
le e
mp
loym
en
t
Occupational, academic, and life skills
The Basic Career Pathways Model
8
Program Inputs: Signature CP Service Strategies Comprehensive assessment
– Academic and non-academic skills
Basic and technical skills instruction– Modularization, contextualization, acceleration, flexible delivery,
active learning
Supports – Pro-active advising and guidance, supplemental instruction,
social supports, supportive services, financial assistance
Employment connections– During and after training
Foundational Academic SkillsFoundational Academic Skills
Intermediate Outcomes
Increase Performance & Persistence in Training
• Certificate/Diploma• 2- year, 4- year Degree
Improve Performance & Advancement in Jobs
• Earnings• Benefits• Job security
Improve Other Outcomes
• Income & assets• Child & adult well being• Local economic growth
Participant Characteristics
•Demographic•Educational•Economic
Program Inputs Primary Outcomes
TO
NE
XT
ST
EP
Occupational SkillsOccupational Skills
Psycho-social Factors Psycho-social Factors
Career Orientation and Knowledge Career Orientation and Knowledge
Resource Constraints Resource Constraints
Other Personal and Family ChallengesOther Personal and Family Challenges
Contextual Factors: Institutional, Economic, SocialContextual Factors: Institutional, Economic, Social
Initial Targeting & Placement Decisions
Initial Targeting & Placement Decisions
Theory of Change for Career Pathways
Illustration #1: Pima Community College Pathways to Healthcare Program (Arizona)• Basic elements
– In partnership with county One-Stop, college provides training and credentials in 16 health care occupations, organized into 5 broad pathways (+ 3 broad levels), programs varying from 1-24 months. Targets low-income adults, substantially Latino population. Funded by federal HPOG program.
Signature strategies– Instruction: Upfront assessment and training planning, w/10-week
contextualized basic skills development program if below required entry skill levels. Many programs offer streamlined/compressed formats, translate “clock-hours” at lower levels to prerequisites to higher levels.
– Supports: More intensive and coordinated case management (via One-Stop) and academic advising and services (via college), financial support covers training and related expenses.
– Employment: Outreach to engage local employers in training, internships, employment; use One-Stop services; MOUs linking to various county and state agencies.
10
Illustration #2: Year Up (National)• Basic elements
– National non-profit operating in 8 urban areas. Partners with community colleges and major employers in financial services and information technology. Targets disadvantaged youth, 18-24, with high school diploma/equivalent.
Signature strategies– Instruction: Six-month “learning development” phase provides customized
training in technical and contextualized basic and professional skills (e.g., communication, behavior). Earn 14+ college credits through agreement w/local colleges.
– Supports: “High support, high feedback” environment—learning communities, pro-active counseling, structured group sessions, supportive services, and weekly performance-based stipend of up to $260 throughout the year. Post-program follow-up and engagement.
– Employment: Second six-month internship phase places students in entry-level financial services/IT positions with major employers, who contribute about half of total program costs per participant.
11
Illustrative Evidence on More Explicit Career Pathways Approaches: Comprehensive, well-targeted “first step” training programs can have substantial impactsSectoral training (Maguire et al. 2010)•3 experienced CBOs provide customized short-term training in varied high-demand fields•Disadvantaged adults, HS+, careful screening (n=1,014)
Year Up (Roder & Elliott 2011)•National org provides 6 months customized training + 6 month paid internship in IT and finance sectors•Disadvantaged urban youth 18-24, HS+•Program articulates w/local colleges, completers earn 14+ credits•Results from small initial experiment (n=164)
12
Examples of Recent Experimental Findings on Signature CP Strategies
Learning communities– As bridge program (Fein et al. 2006)– By linking courses (Sommo et al. 2012; Visher et al. 2012)
More intensive, specialized personal guidance and support– Coaching (Bettinger & Baker 2011)– Help with financial aid applications (Bettinger et al. 2012)– Student success courses + support (Weiss et al. 2011)
Increased financial support– Performance-based scholarships (Patel & Richburg-Hayes 2012)– Unconditional needs-based grants (Goldrick-Rab et al. 2011)
Psycho-social interventions– E.g., implicit theories of intelligence, normalizing worries about fitting in and
success (Yeager & Walton 2011)
13
Designing Evaluations that Matter
Evaluations should be designed in ways that invite practitioners to make use of the results and adopt solid practices based on evidence. We need to recognize the role of motivated program leaders at the center of evaluation efforts, to ensure that these efforts advance program theory and practice, rather than merely fulfilling a funder request. Research tells us that organizations and their leaders need to own and trust information in order to use it. Partnership on the ground between practitioners and evaluators, along with the long-term support of committed public and private funders, is indispensable if the goal is to deliver evaluations that actually improve program quality and effectiveness.
– Public/Private Ventures, Priorities for a New Decade: Making (More) Social Programs Work (Better), 2011.
PrinciplesElucidate and address theories of program effectivenessDevelop strong common metrics that will be easily understood and widely acceptedDesign to anticipate interest in scaling and replicating successful approachesFoster appreciation for technical rigor (e.g., experimental designs)Produce evaluation products useful to policymakers and practitioners
– Emphasis on active interaction: evaluators-practitioner, practitioner-practitioner
14
Example: The ISIS Evaluation
First national random assignment study of U.S. career pathways programs
Separate tests of nine relatively comprehensive career pathways programs, with implementation and cost-benefit studies
Central goals– Assess overall effectiveness of each program– Understand “why and under what conditions?”
• Work within well-specified theory of change• Strong focus on impacts on exposure to program inputs, intervening outcomes,
heterogeneity• Non-experimental analysis of mediators of impacts on primary outcomes
Strong emphasis on collaboration helps ensure findings will count…
15
Collaboration Increases Chances that Findings Will Matter Extensive stakeholder outreach + literature review identify career pathways
focus in ISIS… Early and ongoing exchanges foster interest and create positive climate for
experiments among potential sites– Working w/sites as partners w/complementary expertise and interests– Building an ongoing “learning community”
• Learn from experts on the team, peer-to-peer exchange, participation in external forums
• Help plan and execute dissemination strategies
Maintain and extend relationships with wider stakeholder network– Including private and public funding agencies, policy makers, professional associates,
advocates, practitioners, – Through meetings, website, public webinars, varied publications– On diverse interests: related projects, substantive topics/directions in ISIS, evaluation
findings, evaluation products,
So that findings aren’t limited to a small number of reports, but shared in dynamic, diversified ways with engaged audiences
16
Summary
Career pathways framework useful in program and evaluation design, organizing emerging findings
Genuine partnerships with practitioners and other stakeholders key to designing evaluations that matter
Thank you!
17