summary report evaluation of the cysa/tanf program in california
DESCRIPTION
Summary Report Evaluation of the CYSA/TANF Program in California. PADM 522 Presentation – November 20 th , 2012. Doug Calderwood Edwin Kitzes Cliff Howard Stoycho Ivanoff. Evaluation of F ederal A ssistance. - PowerPoint PPT PresentationTRANSCRIPT
Summary ReportEvaluation of the CYSA/TANF
Program in California
Doug CalderwoodEdwin KitzesCliff Howard
Stoycho Ivanoff
PADM 522 Presentation – November 20th, 2012
Evaluation of Federal Assistance• Chief Probation Officers of California sponsored an
evaluation of the implementation and impact of California Youth Services Act (CYSA) distribution of federal assistance administered the Temporary Assistance for Needy Families (TANF) to California county probation departments (CPDs).
• 3 year Evaluation conducted by the RAND corporation.
Howard
Four Goals of Evaluation• provide timely feedback to Chief
Probation Officers of California on CYSA/TANF implementation,
• document how county probation departments (CPDs) have used CYSA/TANF funds
• assess impact of local interventions, and • draw lessons for improving overall
program design and operation
Howard
Evaluation Sought Answers to 5 Questions
1. What programs were implemented?2. What CYSA/TANF services were provided?3. What CYSA/TANF services and programs were
being provided in the juvenile halls and camps/ranches?
4. What was the impact of CYSA/TANF at the individual and system level?
5. What was the CYSA/TANF funding environment and what were county claiming experiences?
Howard
Probation Departments Administered at the County Level – 58 Counties
Howard
California County Probation Department Funding
State and Federal Funds
Howard
California funding for Probation Facilities
• California county probation departments (CDPs) started receiving federal funds under Title IV-A-EA of the Social Security Act in July 1993 to reimburse institutional costs for juvenile camps and ranches and many child welfare services.
• Ended on December 31, 1995 with the passage of Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) (a.k.a. welfare reform act)
Howard
California County Probation Departments State and Federal Funding
Howard
Sequence of federal assistanceTitle IV-A-EA to TANF
Title IV-A-Emergency Assistance Jul 1993 -Dec 1995PRWORA TANF July 1997
Howard
TANF Replaces Title IV-A-EA Funding• PRWORA instituted Temporary Assistance to
Needy Families (TANF) which became effective July 1, 1997
• TANF is a block grant to states to target needy families – it replaced previous welfare programs
• TANF for California County Probation Departments was around $168M annually during the time of this study.
• CPDs went for 18 months w/o Title IV-A-EA federal assistance
Howard
Intent of CYSA• Provide a continuum of family-focused, case-
specific services in a community-based setting that addresses the full spectrum of child and family needs, including services provided in county-operated residential care facilities
• Programs were to focus more on prevention/intervention when possible
Howard
Goals of CYSA• Legislation had 3 main goals– Keep probation youths from further crime– Help probation and at-risk youths to develop skills to avoid
dependence on public assistance– Help achieve 4 overarching federal TANF goals
• Provide assistance to families so youths may be cared for in their homes• Reduce dependence of needy parents on government
benefits by promoting job preparation, work and marriage• Encourage formation/maintenance of two-parent families• Prevent/reduce incidence of out-of-wedlock pregnancies
Howard
CYSA Program Planning Procedure• Establish local planning council to advise the county Chief
Probation Officer including agencies– County departments of health, mental health, probation, child
protective services and education– Local school districts– City/county law enforcement agencies– Community organizations that serve at-risk youth– One or more youth who are at-risk– Parents of family of at-risk youth
• Chief Probation Officer prepares the written plan• County Board of Supervisors must approve the CYSA/TANF
programs
Howard
5 Key Principles CYSA/TANF• be oriented toward principles of personal
responsibility and self-reliance• use available community resources to the
extent possible• be based on case plans that consider
family concerns, priorities, and resources• be family-focused• address identified immediate needs, as
well as underlying risk factors contributing to problems that are more pervasive and recurrent in nature
Howard
CYSA/TANF Goals pursued by providing a spectrum of 23 authorized services
CYSA/TANF Services Areas Emergency shelter Parent peer support
After-care servicesFamily crisis intervention Parenting skills
Anger management Family mentoring Pre-vocational trainingAvailability of community services Gang intervention Respite careCase management Home detention Sex/health educationCounseling, monitoring, and treatment
Individual, family, group counseling
Social responsibility training
Drug/alcohol education Life skills counseling
Therapeutic day treatment
Educational advocacyMental health assessment
Transportation to needed services
Howard
Spectrum of Juvenile Cases5 categories from least to most serious
• At risk youth – failing academically, poverty, disabled, disadvantaged, certain ethnic minorities
• WIC * 601 – accused of status offense (an offense that’s not applicable to adult)
• WIC * 602 – accused of misdemeanor or felony• Wardship – under court supervision, placed at home
or at a relative’s house on probation • Custody – under court supervision, sentenced from
0-12 months in juvenile camp or ranch* WIC - California Welfare and Institutions Code
Howard
Change Model from Chen
Intervention Determinants Outcome23 authorized service categories such as: • After-care services• Anger management• Availability of community
services• Case management• Counseling, monitoring,
and treatment• Drug/alcohol education• Educational advocacy• Emergency shelter• Family crisis intervention• Family mentoring• Gang intervention• Home detention
Function of Juvenile Case - Includes things like:• Home/shelter conditions• Family/home conditions• Exposure to:o Drugs/alcoholo Gang violenceo Physical vulnerabilitieso Verbal abuse
• Coping abilities• Decision making abilities• Access to harm others• Academic success level• Food availability
Function of Case –Goals include:• Recovery from
transgression• Avoid repeat offense• Live at home• Not dependent on
public assistance• Develop Job skills• Stable 2-parent families• Avoid out-of-wedlock
pregnancies• Protect community• Academic success
Howard
Review of Evaluation Scope,Program Goals, Authorized Services
• CYSA has 6 goals• California has 58 counties• Legislation authorizes 23 service categories• Juvenile cases span at least 5 broad categories of
seriousness (at-risk youth to detention facilities)• Service centers include multiple government and
community facilities and organizations • CYSA/TANF funding is combined with several other
county, state, and federal sourcesHow do you evaluate this?
Howard
Four Components of Evaluation• statewide implementation surveys fielded to
all 58 counties (Years 1 and 3)• an 11-county process study (Years 1 and 2)• an outcomes/impact analysis (Years 1 and 2)• a funding/claim submission analysis (Year 3)
5 questions to be addressed by evaluation1. What programs were implemented?2. What CYSA/TANF services were provided?3. What CYSA/TANF services and programs were being provided in the
juvenile halls and camps/ranches?4. What was the impact of CYSA/TANF at the individual and system level?5. What was the CYSA/TANF funding environment and what were county
claiming experiences?Howard
Data Collected in the Evaluation• Programs implemented under CYSA/TANF (Cliff)• CYSA/TANF services provided (Edwin)• CYSA/TANF services provided in the juvenile halls
and camps/ranches (Edwin)• The impact of CYSA/TANF at the individual and
system levels (Doug)• The CYSA/TANF funding environment and
experiences (Doug)• Summary, Conclusions, and Observations
(Stoycho)Howard
Distribution of Program Funds Along the Continuum of Options
Programs ImplementedHoward
custody
multiplemultiple
custody
Distribution of Program Funds Along Target Population
601 truan
ts
601 runaway
adults
Programs ImplementedHoward
602s
Percent of Counties with Programs That Target Youths and Families Along the Continuum of Options (N=55)
Programs ImplementedHoward
Key Programs in 6 Various Counties
Juvenile assessment centerFamily services program
Personal/social responsibility Juvenile Hall programCamps servicesVoucher program
Medium-level custody programJuvenile Hall Youth Center ServicesBoot camp program
Youth and family resources centersJuvenile Hall unitDiversion program for at-risk youths
Multiagency comprehensive community-based programAftercareDrug Court
Prevention network
Programs Implemented
County B
County E
County D
County G
County C
County F
Howard
Services Areas Categorized into Four Groupings
Formal Treatment Life Skills DevelopmentCounseling, monitoring, treatmentIndividual, family counselingDrug/alcohol educationMental health assessmentTherapeutic day treatmentMental health assessmentFamily crisis intervention
Family mentoringAnger managementLife skills counselingSex/health educationGang interventionParenting skillsEducational advocacy
Coordination OtherAvailability of community servicesCase managementTransportation to needed service
Home detentionAfter-care servicesEmergency shelter
Howard
Types of CYSA/TANF Services Outsourced (N=24)
Programs Implemented Howard
Number of Youths and Families Served in 44 Reporting Counties (June 2002) (N=44)
HowardPrograms Implemented
CYSA/TANF Mediating Variables• Implemented during a time when juvenile
programs were being overhauled• Comprehensive integrated planning was
encouraged• Multiagency planning bodies existed already• Needs assessments had already been done• Built on existing CPD relations with service
providers
Programs ImplementedHoward
CYSA/TANF Program Moderators
• Difficulty to implement multiservice delivery model• Turf concerns, philosophical differences, conflicts
between service agencies• Procedures for claiming funds time consuming• CYSA implementation guides were vague• Difficult to evaluate qualifications of some service
providers• Difficult to write performance-based contracts• Staff shortages in early years
Programs ImplementedHoward
CYSA/TANF Services Provided
Edwin Kitzes
Services ProvidedKitzes
From the Chen Reading• Assessment-Oriented Process Evaluation• Primary for External Stakeholders to
evaluate how well a program is being implemented
• Evaluate three main program purposes:1) Meeting Accountability Needs2) Meeting Program Improvement Needs3) Provide Interpreting Outcome
Evaluation
Services Areas Categorized into Four Groupings
Formal Treatment Life Skills DevelopmentCounseling, monitoring, treatmentIndividual, family counselingDrug/alcohol educationMental health assessmentTherapeutic day treatmentMental health assessmentFamily crisis intervention
Family mentoringAnger managementLife skills counselingSex/health educationGang interventionParenting skillsEducational advocacy
Coordination OtherAvailability of community servicesCase managementTransportation to needed service
Home detentionAfter-care servicesEmergency shelter
Howard
Services ProvidedHow was the data gathered?
• Year 1 Survey – before/after – all programs
• Year 3 Survey – Counties primary programs/services provided
• 58-Counties• Compare pre- and post-CYNA/TANF• Compare different size counties, big vs.
small• ‘Real Change’ vs. adjustments of claims
Services ProvidedKitzes
58 Counties in California
Howard
Formal Treatment Services
46 Counties 12 Counties
Before | After
Services ProvidedKitzes
Coordination Services
Services ProvidedKitzes
Life Skills Development
Services ProvidedKitzes
Other Services
Services ProvidedKitzes
Services ProvidedSummary
• Majority of funds used to ‘Enhance existing services’
• Larger counties were more likely to enhance service vs. smaller counties which used funds to add new services.
Kitzes
Services ProvidedYear 3
Services ProvidedKitzes
Services Provided
Services ProvidedKitzes
Types of CYSA/TANF Services Outsourced (N=24)
Programs Implemented Howard
Spectrum of Juvenile Cases6 categories from least to most serious
• At risk youth – failing academically, poverty, disabled, disadvantaged, certain ethnic minorities
• WIC * 601 – accused of status offense (an offense that’s not applicable to adult)
• WIC * 602 – accused of misdemeanor for felony• WIC* 777 – accused of probation violation• Wardship – under court supervision, placed at home or at a
relative’s house on probation • Custody – under court supervision, sentenced from 0-12
months in juvenile camp or ranch* WIC - California Welfare and Institutions Code
Howard
Services Provided
Services ProvidedKitzes
Services ProvidedYear 3 Summary
• 11 of the 23 services were provided in more than 75% of the counties.
• There was “real change” made in the delivery of the services due to CYSA/TANF funds - virtually unanimous in all counties.
• Majority of funds used to ‘Enhance existing services’• Larger counties were more likely to enhance service
vs. smaller counties which used funds to add new services.
Kitzes
CYSA/TANF Services Provided inJuvenile Halls and Camps/Ranches
Services in halls/campsKitzes
Juvenile Hall, Orange County
Juvenile Hall, San Deigo County
http://www.youtube.com/watch?v=QTI2u3bmKFQ
Juvenile Halls and Camps/Ranches
http://www.youtube.com/watch?v=gyaO634JWpU
Services ProvidedHow was the data gathered?
• Year 2 Survey – In-depth survey of 11 Counties
• Purpose: gather general information: to gather information on services provided in Halls/Camps/Ranches
• And results of CYSA/TANF funds‘Real Change’ vs. adjustments of claims
• Compare pre- and post-CYNA/TANFServices Provided
Kitzes
Services provided in the juvenile halls and camps/ranches
Services in halls/campsKitzes
1,50026
Services provided in the juvenile halls and camps/ranches
Services in halls/campsKitzes
KitzesServices in halls/camps
Services provided in the juvenile halls and camps/ranches
Services ProvidedYear 2 Summary Halls
• Relative short stays in juvenile halls; often occupied at daily capacity.
• Claiming strategies were driven to maximize Federal revenues received.
• There was “real change” made in the delivery of the services due to CYSA/TANF funds.
• Formal treatment, Life Skills Development, but few Coordination and Other Services
• Majority of funds used to ‘Enhance existing services’
Kitzes
Services ProvidedYear 2 Summary Camps/Ranch
• Mirrored Juvenile Hall results with the exception of more services provided
• AGAIN, • Claiming strategies were driven to maximize
Federal revenues received.• There was “real change” made in the delivery
of the services due to CYSA/TANF funds.
Kitzes
CYSA/TANF
Doug Calderwood
Individual and Systemic Impact
Funding Environment and Experiences
CYSA/TANF Systemic and Individual Impact
• System Evaluation– Year 1 and 3 surveys– Chiefs’ assessments– Evaluation Team’s
assessment • Individual Evaluation– Four County Programs’
outcomes
Calderwood
The Multi-Goal, Theory-Driven Approach to Evaluation
Huey T. Chen & Peter H. Rossi• In one field after another, evaluation
researchers find that the programs in place or contemplated have few or no effects of the sort intended by their designers.
(1980, p. 106)
Calderwood
“…[S]tatements of program goals tend to be vague and/or stated in terms that do not lend themselves easily to precise measurement. …[I]n practice one is able to define only a few measurable, specific outcome variables (goals) …typically quite narrow compared to the original social problem that the social program aims to alleviate.”
(Chen & Rossi, 1980, p. 109)
Calderwood
“Clearly the problems with the conventional paradigm are not to be remedied by using rigorous research designs since there would be even less chance of finding effects using more powerful designs.”
(Chen & Rossi, 1980, p. 109)
Calderwood
System—Year 1 and 3 Surveys
• “CYSA Legislation hoped to achieve specific outcomes” (Turner, Davis, Steinberg & Fain, p. 51)– Effects on juvenile justice services (Table 5.1)– Moderate to large impact (2.9 to 3.9 out of 5)
• “CYSA Legislation’s goal was to…avoid dependency…and reduce…future criminal activity” (Turner et. al., p. 52)– Effects on target population (Table 5.2)– Positive to extremely positive impact (3.4/5 to 4.6/5)
Calderwood
System—Chief’s Assessment• Separate survey of Chiefs in year 3 included
two questions:– Additional activities enabled?• Addition and enhancement of specific services• Base funding for custody programs• Expansion of range of services
– Impact if funding went away?• Services would be cut
Calderwood
System—Evaluation Team
• CYSA/TANF:– Filled an important funding gap– Supported a shift from suppression,
enforcement and monitoring to rehabilitation and therapy
– Increased system-wide collaboration and planning
– Enabled viability of probation in county level children’s programs
Calderwood
Individual—Four County Programs
• Research criteria for selection– Distinct CYSA/TANF Programs – Comparison group available– Identifiable CYSA/TANF clients– County collects client characteristics, services
delivered and outcomes• Inclusivity selection criteria– Counties: Northern/Southern and Large/Small– Programs: Field/Institutional
Calderwood
County K: Adult TANF Caseload Program
• TANF-eligible parolees received added family services
• Case load limits excluded some eligible—the control group
• TANF and control had equal risks, TANF had more strengths (housing, family stability and food)
• TANF had fewer probation violations, more technical violations
Calderwood
County F:Juvenile Assessment Center Program
• Assessment, case planning and follow-up– 2,437 youths served from 2/98 to 5/02
• Historical control group – 787 youths from ’97
• Recidivism was less for program participants at 6, 12 and 18 months
• Time before first subsequent referral was also longer
Calderwood
County H:Placement Readiness Recidivism Program
• Youth with mental illness in custody got weekly independent therapy, theme groups and daily recreation
• 200 youths participated from 6/99 to 3/02• Historical control group of 200 youths who
would have been eligible• Participants has lower recidivism but worse
behavior while in custody
Calderwood
County D:Youth and Family Resource Centers
• Day-reporting and treatment centers for “8%” youth—a profile of a small proportion accounting for 50% of repeat offenses
• Control group met profile but were not assigned
• Results showed little difference for those in the program
Calderwood
CYSA/TANFFunding Environment and Experiences
• Probation Funding Environment
• Claiming Strategies
• Leveraging Strategies
• Fiscal Pressures
Calderwood
California County Probation Departments State and Federal Funding
Howard
Funding Environment
• CPD funding is about 10% CYSA/TANF (more in small counties)
• Title IV-E funds can’t be used in custody settings, so TANF is more heavily claimed there.
• This is part of the difficulty evaluating “pure” CYSA/TANF programs
Calderwood
Funding Environment
• About 1/3 of counties experienced fiscal pressure—withholding or re-appropriation of CYSA/TANF funds
• Chiefs of CPDs felt CYSA/TANF funding was extremely important overall
• CYSA/TANF money replaced lost Title IV-A-EA funds
Calderwood
CYSA/TANF Program Complications
• 23 services in four broad categories• 58 counties setting up programs• CYSA seeks to bridge TANF and Juvenile Justice• TANF funding often used to cover programs
defunded when Title IV-A-EA went away• It is unlikely that local programs have clear
program theories• Programs have little traditionally measurable
effectCalderwood