Developing a Program of Postsecondary Developing a Program of Postsecondary Academic Instruction Over the Transforming Academic Instruction Over the Transforming

Lives NetworkLives Network

Improving Evidence of Impact through a National Study of Improving Evidence of Impact through a National Study of

Postsecondary Academic Programming in State prisonsPostsecondary Academic Programming in State prisons

Dr. Stephen Meyer, Principal InvestigatorDr. Stephen Meyer, Principal InvestigatorRMC Research CorporationRMC Research Corporation

Cindy Borden & Penny Richardson, Field InvestigatorsCindy Borden & Penny Richardson, Field InvestigatorsNorthstar Correctional Education ServicesNorthstar Correctional Education Services

Dr. Stephen Steurer, Project DirectorDr. Stephen Steurer, Project DirectorCorrectional Education AssociationCorrectional Education Association

CEA Leadership Forum March 10, 2008CEA Leadership Forum March 10, 2008

This SessionThis Session

• Research in Correctional EducationResearch in Correctional Education• Promotion of Rigorous Research in EducationPromotion of Rigorous Research in Education• Current State of Correctional Postsecondary Current State of Correctional Postsecondary

Academic ProgrammingAcademic Programming• College of the Air and the Transforming Lives College of the Air and the Transforming Lives

NetworkNetwork• Overview of the StudyOverview of the Study• More Detailed Information about Incentives, More Detailed Information about Incentives,

Data Collection Activities, Timeline, and Data Collection Activities, Timeline, and Expectations of Participating SitesExpectations of Participating Sites

Research in Correctional Research in Correctional EducationEducation

Example: Chappell (2004)Example: Chappell (2004)

• Review of Postsecondary CE and Review of Postsecondary CE and Recidivism Studies Conducted in the 1990sRecidivism Studies Conducted in the 1990s

• Found a correlation (.31) between PSCE Found a correlation (.31) between PSCE and recidivismand recidivism

• Only 15 studies met criteria for inclusion. Only 15 studies met criteria for inclusion. These studies had substantial limitations:These studies had substantial limitations:– Selection bias; control groups for only 3 studiesSelection bias; control groups for only 3 studies– Failure to distinguish between secondary and Failure to distinguish between secondary and

postsecondarypostsecondary– Different definitions of PSCE and recidivisimDifferent definitions of PSCE and recidivisim

Chappell, C.A. (2004). Post-Secondary Correctional Education and Recidivism: A Meta-Analysis of Research Conducted 1990-Chappell, C.A. (2004). Post-Secondary Correctional Education and Recidivism: A Meta-Analysis of Research Conducted 1990-1999. 1999. Journal of Correctional EducationJournal of Correctional Education , June., June.

Example: Wilson, Gallagher, & Example: Wilson, Gallagher, & MacKenzie (2000)MacKenzie (2000)• Meta analysis of 33 experimental and quasi-Meta analysis of 33 experimental and quasi-

experimental evaluations of education, experimental evaluations of education, vocation, and work programs. vocation, and work programs.

• Found that program participants recidivated Found that program participants recidivated at a lower rate than nonparticipantsat a lower rate than nonparticipants

• ““The generally weak methodological The generally weak methodological character of these studies, however, prevents character of these studies, however, prevents attributing this observed effect on criminal attributing this observed effect on criminal behavior to the activities of the programs.” behavior to the activities of the programs.”

Wilson, D.B., Gallagher, C.A., & MacKenzie, D.L. (2000). A Meta-Analysis of Corrections-Based Education, Vocation, and Wilson, D.B., Gallagher, C.A., & MacKenzie, D.L. (2000). A Meta-Analysis of Corrections-Based Education, Vocation, and Work Programs for Adult Offenders. Work Programs for Adult Offenders. Journal of Research in Crime and DelinquencyJournal of Research in Crime and Delinquency , 37(4)., 37(4).

Research in CE: LimitationsResearch in CE: Limitations

• Post-treatment outcomesPost-treatment outcomes

• Recidivism as primary outcomeRecidivism as primary outcome

• Small study samplesSmall study samples

• Quasi-experimental designsQuasi-experimental designs

• Limited statistical controlsLimited statistical controls

• Limited measurement of program Limited measurement of program componentscomponents

See Lewis, J. (2006). Correctional education: Why it is only promising. See Lewis, J. (2006). Correctional education: Why it is only promising. Journal of Correctional EducationJournal of Correctional Education , 57(4)., 57(4).

Promotion of Rigorous Promotion of Rigorous Research in EducationResearch in Education

Policy Emphasis on Rigorous Policy Emphasis on Rigorous Research Research

• Focus on evidence-based policy and Focus on evidence-based policy and practicepractice– Comprehensive School Reform Demonstration Comprehensive School Reform Demonstration

program in 1990sprogram in 1990s– No Child Left Behind (2002)No Child Left Behind (2002)

• Requires scientifically valid and readily Requires scientifically valid and readily interpretable syntheses of research on interpretable syntheses of research on practical and replicable programs and practical and replicable programs and policiespolicies

What Works ClearinghouseWhat Works Clearinghouse

• Established by the US Department of Established by the US Department of Education to be a central source of Education to be a central source of scientific evidence of what works in scientific evidence of what works in education education

• Conducts reviews that include only Conducts reviews that include only studies that meet standards for studies that meet standards for scientific evidencescientific evidence

What Works Clearinghouse What Works Clearinghouse ConsiderationsConsiderations• Relevance to the reviewRelevance to the review

– Were the intervention and outcome measures properly Were the intervention and outcome measures properly defined?defined?

• Generality of findingsGenerality of findings– Was the intervention tested on relevant participants and Was the intervention tested on relevant participants and

environments?environments?

• Precision of outcomesPrecision of outcomes– Could accurate effect sizes be derived from the study Could accurate effect sizes be derived from the study

report?report?

• Clarity of causal evidenceClarity of causal evidence– Was the intervention the cause of the change in the Was the intervention the cause of the change in the

outcome? outcome?

What Works Clearinghouse What Works Clearinghouse Study EligibilityStudy Eligibility

•To be eligible for WWC review, a study must be a randomized controlled trial or an eligible quasi-experiment.– Random assignment=“gold standard” of

scientifically based research– Comparison groups=“silver standard”

Rigorous Criteria Result in Few Rigorous Criteria Result in Few Studies for Review - ExampleStudies for Review - Example

• Review of Review of over 1,300over 1,300 studies that studies that examined the effect of teacher examined the effect of teacher professional development on student professional development on student achievement (Yoon et al., 2007)achievement (Yoon et al., 2007)

• Found that Found that only nineonly nine met What met What Works Clearinghouse standards for Works Clearinghouse standards for rigorous evidence!rigorous evidence!

Yoon, K. S., Duncan, T., Lee, S. W.-Y., Scarloss, B., & Shapley, K. (2007). Yoon, K. S., Duncan, T., Lee, S. W.-Y., Scarloss, B., & Shapley, K. (2007). Reviewing the evidence on how teacher professional development affects student achievement Reviewing the evidence on how teacher professional development affects student achievement (Issues & Answers (Issues & Answers Report, REL 2007–No. 033). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Report, REL 2007–No. 033). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Southwest. Educational Laboratory Southwest.

Promoting More Rigorous Promoting More Rigorous ResearchResearch

• The Institute of Education Sciences (IES)The Institute of Education Sciences (IES)– Established in 2002 to “provide rigorous Established in 2002 to “provide rigorous

evidence on which to ground education evidence on which to ground education practice and policy”practice and policy”

– Has the goal of “identifying, developing, and Has the goal of “identifying, developing, and validating effective education programs, validating effective education programs, practices, policies, and approaches as well as practices, policies, and approaches as well as understanding the factors that influence understanding the factors that influence variation in their effectiveness such as variation in their effectiveness such as implementation.”implementation.”

IES 2007 GrantsIES 2007 Grants

• Postsecondary Education added as Postsecondary Education added as new research topicnew research topic

• Only three proposals funded in FY Only three proposals funded in FY 2007 (from over 50 applications)2007 (from over 50 applications)

IES Priorities for Study DesignIES Priorities for Study Design

• Clearly defined interventionsClearly defined interventions

• Random Assignment – Cluster Randomized Random Assignment – Cluster Randomized Trials Trials

• Study design and sample size that Study design and sample size that provides adequate statistical powerprovides adequate statistical power

• Process (implementation) measuresProcess (implementation) measures

Benefits of Random Benefits of Random AssignmentAssignment

•Participants have equal chances of being assigned to each of the conditions

•Differences between conditions cannot be attributed to pre-existing differences between participants and/or selection biases

•Provides evidence of causality

Benefits of This StudyBenefits of This Study• Results are expected to provide valuable Results are expected to provide valuable

information about the effectiveness and information about the effectiveness and impact of a widely available postsecondary impact of a widely available postsecondary academic delivery model that can increase academic delivery model that can increase access, persistence, and completion of access, persistence, and completion of postsecondary education by incarcerated postsecondary education by incarcerated youth. youth.

• Because access to this type of curriculum by Because access to this type of curriculum by incarcerated youth is limited, development incarcerated youth is limited, development of this model and documentation of its of this model and documentation of its impact has great potential to improve impact has great potential to improve postsecondary correctional education. postsecondary correctional education.

• Study design will provide evidence that Study design will provide evidence that meets WWC standardsmeets WWC standards

State of PS Academic State of PS Academic Programming and College Programming and College

of the Air and the of the Air and the Transforming Lives Transforming Lives

NetworkNetwork

Current State of Correctional Current State of Correctional Postsecondary Academic Postsecondary Academic ProgrammingProgramming

• Programs on rise, recovering from loss of Programs on rise, recovering from loss of Pell grantsPell grants

• IYO age/funding cap legislative changesIYO age/funding cap legislative changes• 60% state/73% federal offenders have 60% state/73% federal offenders have

earned pre-requisite HSD/GEDearned pre-requisite HSD/GED• 5% nationwide have access to academic 5% nationwide have access to academic

post secondarypost secondary• Slow migration from vocational to academicSlow migration from vocational to academic• Politically unpopularPolitically unpopular• Lack of funding primary barrierLack of funding primary barrier

College of the Air and the College of the Air and the Transforming Lives NetworkTransforming Lives Network

• Nationwide satellite broadcast owned and Nationwide satellite broadcast owned and operated by CEAoperated by CEA

• Three 22-credit certificates culminating in Three 22-credit certificates culminating in A.A.A.A.

• $325/three-credit course (no out-of-state $325/three-credit course (no out-of-state tuition) offered by Milwaukee Area tuition) offered by Milwaukee Area Technical CollegeTechnical College

• 200 level or above for transferability200 level or above for transferability

• $1195 annual site fee for unlimited TLN use$1195 annual site fee for unlimited TLN use• Pioneered by Wisconsin DOCPioneered by Wisconsin DOC

Overview of the IES StudyOverview of the IES Study

Overview of the IES StudyOverview of the IES Study

• The College of the Air curricula delivered The College of the Air curricula delivered via the Transforming Lives Network via the Transforming Lives Network (COA/TLN) has great potential to increase (COA/TLN) has great potential to increase access, persistence, and completion of access, persistence, and completion of courses by incarcerated youth leading to courses by incarcerated youth leading to postsecondary degrees. postsecondary degrees.

• This study is designed to obtain impact This study is designed to obtain impact data to determine the efficacy of this data to determine the efficacy of this approach. approach.

Research QuestionsResearch Questions

1.1. To what extent does College of the Air delivered via the To what extent does College of the Air delivered via the Transforming Lives Network (COA/TLN):Transforming Lives Network (COA/TLN):

a.a. Increase rates of Increase rates of participationparticipation in postsecondary and in postsecondary and other academic programming?other academic programming?

b.b. Improve participants’ Improve participants’ academic achievementacademic achievement outcomes outcomes and and progress toward postsecondary academic degreesprogress toward postsecondary academic degrees??

c.c. Improve participants’ Improve participants’ achievement motivationachievement motivation and and educational aspirationseducational aspirations??

d.d. Affect Affect post-release employabilitypost-release employability for participants? for participants?e.e. Affect Affect institutional outcomesinstitutional outcomes, such as institutional , such as institutional

climate and recidivismclimate and recidivism**??2.2. To what extent do aspects of the COA/TLN curriculum To what extent do aspects of the COA/TLN curriculum

and its delivery, institutional support, participant and its delivery, institutional support, participant engagement, and participant characteristics affect engagement, and participant characteristics affect outcomes? outcomes?

*Defined as reincarceration as a result of a new crime or parole violation.*Defined as reincarceration as a result of a new crime or parole violation.

Logic ModelLogic Model

Mediating Factors

Instructional

Program Characteristics

Institutional

Characteristics Participant

Academic Engagement

Participant

Characteristics

Participant Outcomes

Increased Academic Achievement Increased Achievement Motivation Increased Educational Aspirations Progress toward Postsecondary

Degree Improved Employability and

Employment (post-release)

Participation in . . . 1. College of the Air

via the Transforming Lives Network (COA/TLN)

2. Other forms of

Postsecondary Academic Programming

Institutional Outcomes Increased Participation in

Academic Education Programs Improved Institutional Climate Reduced Rates of Recidivism

TimelineTimeline

• Three-year study, beginning fall Three-year study, beginning fall 20082008

• Fall and spring data collection Fall and spring data collection through spring 2011through spring 2011

SampleSample• Forty-four prisons that serve a high Forty-four prisons that serve a high

concentration of youth offendersconcentration of youth offenders– Have the infrastructure in place to provide Have the infrastructure in place to provide

postsecondary academic instructionpostsecondary academic instruction– Have not offered COA/TLN in the pastHave not offered COA/TLN in the past– Provide IYO or other funds for PS academic programsProvide IYO or other funds for PS academic programs– Willing to be randomly assigned to treatment or Willing to be randomly assigned to treatment or

control condition as part of the studycontrol condition as part of the study– Have a population that will provided a minimum of Have a population that will provided a minimum of

approximately 15 study participants beginning PS approximately 15 study participants beginning PS academic programming in fall 2008 who are: (1) academic programming in fall 2008 who are: (1) between the ages of 18 and 25; (2) with a release date between the ages of 18 and 25; (2) with a release date between 1 and 5 years; (3) in possession of a high between 1 and 5 years; (3) in possession of a high school diploma or equivalent; and (4) whose tuition school diploma or equivalent; and (4) whose tuition costs are paid using IYO or other grant funding. costs are paid using IYO or other grant funding.

Cluster Randomized TrialCluster Randomized Trial

• Random assignment of prisons within states Random assignment of prisons within states to offer either experimental or control to offer either experimental or control conditioncondition

– If assigned to experimental condition, COA/TLN must If assigned to experimental condition, COA/TLN must be provided as the primary institution-sponsored PS be provided as the primary institution-sponsored PS academic curriculum for the duration of the study. academic curriculum for the duration of the study.

– If assigned to control condition, alternative PS If assigned to control condition, alternative PS academic programming must be made available for academic programming must be made available for the duration of the study. COA/TLN must not be the duration of the study. COA/TLN must not be provided for postsecondary.provided for postsecondary.

– Options for sites with strong PS academic programs Options for sites with strong PS academic programs already in placealready in place

More Detailed Information More Detailed Information about Incentives, Data about Incentives, Data Collection Activities, Collection Activities,

Timeline, and Timeline, and Expectations of Expectations of

Participating SitesParticipating Sites

IncentivesIncentives

• COA/TLN sitesCOA/TLN sites– First Year: Reimbursement of up to $1,500 First Year: Reimbursement of up to $1,500

TLN equipment purchase costs/installation TLN equipment purchase costs/installation and the $1,000 annual site fee. and the $1,000 annual site fee.

– Years 2 and 3: Reimbursement of the $1,000 Years 2 and 3: Reimbursement of the $1,000 annual site fee. annual site fee.

• Control sitesControl sites - $2,500 during the first year and $1,000 each $2,500 during the first year and $1,000 each

subsequent year to be applied toward TLN subsequent year to be applied toward TLN equipment/installation at the conclusion of equipment/installation at the conclusion of the study.the study.

Data CollectionData Collection

• Data collection activities conducted by research Data collection activities conducted by research team at each siteteam at each site

• Informed ConsentInformed Consent

• CAAPCAAP

• College Student SurveyCollege Student Survey

• Site Coordinator SurveySite Coordinator Survey

• Institutional DataInstitutional Data

• College Student Follow-up InterviewCollege Student Follow-up Interview

• Case Studies (interviews, focus groups, Case Studies (interviews, focus groups, observations)observations)

Data Collection: Data Collection: CAAP Critical Thinking Test CAAP Critical Thinking Test

• Upon participant recruitment each Upon participant recruitment each fall (2008, 2009, 2010) and each fall (2008, 2009, 2010) and each subsequent spring through 2011subsequent spring through 2011

• Nationally normed, standardized, Nationally normed, standardized, multiple-choice test that measures multiple-choice test that measures students’ skills in clarifying, students’ skills in clarifying, analyzing, evaluating, and analyzing, evaluating, and extending argumentsextending arguments

Data Collection: Data Collection: College Student SurveyCollege Student Survey

• Upon participant recruitment each fall Upon participant recruitment each fall (2008, 2009, 2010) and each subsequent (2008, 2009, 2010) and each subsequent spring through 2011spring through 2011

• Measures curriculum and instructional Measures curriculum and instructional delivery; academic engagement; delivery; academic engagement; achievement motivation; progress toward achievement motivation; progress toward postsecondary degree; educational postsecondary degree; educational aspirations; institutional support; aspirations; institutional support; institutional climate; respondent institutional climate; respondent characteristics characteristics

Data Collection: Data Collection: Site Coordinator SurveySite Coordinator Survey

• Spring 2009, 2010, 2011Spring 2009, 2010, 2011

• Measures: available postsecondary Measures: available postsecondary programs and inmate participation; programs and inmate participation; eligibility criteria and incentives for eligibility criteria and incentives for participation in postsecondary participation in postsecondary programs; curriculum and instructional programs; curriculum and instructional delivery; institutional support; delivery; institutional support; institutional climate; respondent institutional climate; respondent characteristics characteristics

Data Collection: Data Collection: Institutional DataInstitutional Data• Spring 2009, 2010, 2011Spring 2009, 2010, 2011• Information about: Information about:

– Study ParticipantsStudy Participants – Education – Education background, course and credit completion, background, course and credit completion, sentence length, release date, custody level, sentence length, release date, custody level, transfers, post-release contact information, transfers, post-release contact information, recidivism recidivism

– FacilityFacility - Size, inmate demographics, - Size, inmate demographics, recidivism rates, availability of and recidivism rates, availability of and participation in academic programs, participation in academic programs, educational funding and staffing, education educational funding and staffing, education policiespolicies

– CurriculumCurriculum - Course content, expected time - Course content, expected time commitment, course sequence and commitment, course sequence and credential, delivery mode(s)credential, delivery mode(s)

Data Collection: Data Collection: College Student Follow-Up College Student Follow-Up InterviewInterview• Spring 2009, 2010, 2011Spring 2009, 2010, 2011

• For released inmatesFor released inmates

• Measures of progress toward Measures of progress toward postsecondary degree, postsecondary degree, achievement motivation, achievement motivation, educational aspirations, educational aspirations, employment status, respondent employment status, respondent characteristicscharacteristics

Data Collection: Data Collection: Case Study SitesCase Study Sites

• Random sample of 10 COA/TLN and 5 control sitesRandom sample of 10 COA/TLN and 5 control sites• Site visits during 2008-09, 2009-10, and 2010-11 Site visits during 2008-09, 2009-10, and 2010-11

academic yearsacademic years• Interviews with site coordinators, focus groups with Interviews with site coordinators, focus groups with

participating students, and observation of the participating students, and observation of the learning environment and instructional activitieslearning environment and instructional activities

• Additional information about Additional information about – clarity, value, and relevance of instruction; clarity, value, and relevance of instruction; – factors that facilitate or impede progress; factors that facilitate or impede progress; – factors affecting motivation, course completion; factors affecting motivation, course completion; – postsecondary and career aspirations; postsecondary and career aspirations; – support mechanisms associated with pursuing support mechanisms associated with pursuing

coursework; andcoursework; and– state and local policies affecting participation.state and local policies affecting participation.

TimelineTimeline

Fall 2008

Spring 2009

Fall 2009

Spring 2010

Fall 2010

Spring 2011

Student Survey X X Xa X Xa X CAAP Critical Thinking Test

X X Xa X Xa X

Site Coordinator Survey

X X X

Phone Interview (released students)

X X X

Review of Institutional Records

X X X

Case Study Data Collectionb

X X X

aThe Student Survey and CAAP will be administered in fall 2009 and fall 2010 to newly recruited participants only. bCase study data collection will take place in a random sample of TLN and control sites and will include interviews with site coordinators and students and observation activities.

Participating Sites - Requirements Participating Sites - Requirements

• Have the infrastructure in place to provide Have the infrastructure in place to provide postsecondary academic instructionpostsecondary academic instruction

• Have not offered COA/TLN in the pastHave not offered COA/TLN in the past• Provide IYO or other funds for PS academic Provide IYO or other funds for PS academic

programsprograms• Willing to be randomly assigned to treatment or Willing to be randomly assigned to treatment or

control condition as part of the studycontrol condition as part of the study• Have a population that will provided a minimum Have a population that will provided a minimum

of approximately 15 study participants beginning of approximately 15 study participants beginning PS academic programming in fall 2008 who are: PS academic programming in fall 2008 who are: (1) between the ages of 18 and 25; (2) with a (1) between the ages of 18 and 25; (2) with a release date between 1 and 5 years; (3) in release date between 1 and 5 years; (3) in possession of a high school diploma or possession of a high school diploma or equivalent; and (4) whose tuition costs are paid equivalent; and (4) whose tuition costs are paid using IYO or other grant funding. using IYO or other grant funding.

Participating Sites - Expectations Participating Sites - Expectations

• Site Coordinator for each siteSite Coordinator for each site– Assist with security clearanceAssist with security clearance– Group students for data collection activitiesGroup students for data collection activities– Provide access to student records/ institutional Provide access to student records/ institutional

databasesdatabases– Provide information to track released offendersProvide information to track released offenders

• Two site visits annually Two site visits annually – Fall (August or September)Fall (August or September)– Spring (April or May)Spring (April or May)

• 1-2 day visit; all data collection activities 1-2 day visit; all data collection activities conducted by research staffconducted by research staff

Next StepsNext Steps

• Identification of final sample ASAPIdentification of final sample ASAP• Initial Northstar visit by June Initial Northstar visit by June

• Identify Site CoordinatorIdentify Site Coordinator• Inform administratorsInform administrators• Secure MOU’sSecure MOU’s

• Random assignment of sites in June or Random assignment of sites in June or earlierearlier

Dr. Stephen Meyer, Principal InvestigatorDr. Stephen Meyer, Principal InvestigatorRMC Research CorporationRMC Research Corporation

meyer@rmcdenver.commeyer@rmcdenver.com

Cindy Borden & Penny Richardson, Field InvestigatorsCindy Borden & Penny Richardson, Field InvestigatorsNorthstar Correctional Education ServicesNorthstar Correctional Education Services

northstar@ekit.comnorthstar@ekit.com

Dr. Stephen Steurer, Project DirectorDr. Stephen Steurer, Project DirectorCorrectional Education AssociationCorrectional Education Association

ssteurer@ceanational.orgssteurer@ceanational.org

Contact InformationContact Information