1

Process Evaluation of the Process Evaluation of the King County Family King County Family

Treatment CourtTreatment CourtPreliminary ResultsPreliminary Results

Eric J. Bruns, Tracy JonesEric J. Bruns, Tracy JonesJustin D. Smith, Eric TrupinJustin D. Smith, Eric Trupin

University of WashingtonUniversity of WashingtonDivision of Public Behavioral Health and Division of Public Behavioral Health and

Justice PolicyJustice Policy

March 30, 2006March 30, 2006

2

Components of the Components of the EvaluationEvaluation

Data element and data collection Data element and data collection assessmentassessment

Process evaluationProcess evaluation Interviews with team membersInterviews with team members Comparison study of short-term outcomes Comparison study of short-term outcomes

for KCFTC participants vs. matched for KCFTC participants vs. matched comparison groupcomparison group

Parent interviewsParent interviews Record reviews (KCFTC, TARGET, and CAMIS)Record reviews (KCFTC, TARGET, and CAMIS)

Outcomes evaluation designOutcomes evaluation design Cost-benefit analysis designCost-benefit analysis design

3

Process EvaluationProcess EvaluationKCFTC PROCESSES AND FUNCTIONS•Comprehensive SB Assessment•High quality, appropriate CD Services•Timely/effective MH and other services•Effective care planning and management•Expanded and more frequent visitation•Consistent, timely incentives & sanctions•Random UA Screens•Effective pre-hearing case conferences•Effective judicial interaction

SHORT-TERM OUTCOMES•Eligibility/enrollment completed quickly•Enrollment in appropriate CD services•Parents compliant with/complete treatment•Parents ultimately able to remain sober•Parents/children more fully engaged in svcs•Parents/children receive needed services•Decreased placement disruptions•Parents compliant with court orders•Less negative effect on child well-being•Less disruption of child-parent bonds•Increased family reunification rates•Earlier determination. of alternate placement options

NATIONAL BEST PRACTICES•Communication bw Court and providers•Judge plays active role in Tx process•Judge responds to positive & noncompliant beh.•Mechanisms for shared decision making•Accountability for Tx services•Strategy for responding to noncompliance•MIS allow data to be assembled/reviewed•Enhancement of due process•Team members provided adeq resources

4

MethodMethod Interviews with KCFTC staff and key Interviews with KCFTC staff and key

informants (e.g., advisory group members)informants (e.g., advisory group members) N=35 initially identified; 5 deemed not N=35 initially identified; 5 deemed not

appropriateappropriate N=9 added through staff hires and identification N=9 added through staff hires and identification

by Evaluation Advisory Committeeby Evaluation Advisory Committee Total N=39; 34 completed, 5 scheduled or Total N=39; 34 completed, 5 scheduled or

pendingpending Average interview time = 63 minutes Average interview time = 63 minutes

(range: 40-80 min)(range: 40-80 min)

5

Staff and key informant Staff and key informant interviewinterview

Respondent informationRespondent information General questions about KCFTC goals, General questions about KCFTC goals,

target population, and overall successtarget population, and overall success Eligibility and referral processEligibility and referral process KCFTC process and functionsKCFTC process and functions Adherence to best practices (child welfare Adherence to best practices (child welfare

and treatment courts)and treatment courts) Short-term outcomesShort-term outcomes KCFTC teamwork and collaborationKCFTC teamwork and collaboration Ratings of individual team members’ Ratings of individual team members’

effectivenesseffectiveness Open-ended questions on strengths, Open-ended questions on strengths,

weaknesses, and areas for improvementweaknesses, and areas for improvement

6

RespondentsRespondents

7

Respondents Respondents (total N=34)(total N=34)

Respondent typeRespondent type NN %%

Advisory Group memberAdvisory Group member 99 27%27%

DSHS Social WorkerDSHS Social Worker 44 12%12%

Court Appointed Special Advocate (CASA)Court Appointed Special Advocate (CASA) 44 12%12%

Attorney (Parent or Child)Attorney (Parent or Child) 55 15%15%

JudgeJudge 22 6%6%

CASA Manager or SupervisorCASA Manager or Supervisor 22 6%6%

Social Work Supervisor, Program Social Work Supervisor, Program Coordinator, Family Treatment Court Coordinator, Family Treatment Court Specialist, Wraparound Coordinator, Specialist, Wraparound Coordinator, Treatment Provider, Treatment Provider Treatment Provider, Treatment Provider Supervisor, AAG, OtherSupervisor, AAG, Other

1 Each1 Each 22%22%

Team member interviews conducted 19 (56%)Key informant interviews conducted 15 (44%)

8

Time Spent Directly with Time Spent Directly with KCFTC Court or Families KCFTC Court or Families

(n=33)(n=33)

0%

5%

10%

15%

20%

25%

30%

35%

Percent 30% 21% 18% 3% 27%

<10% 11-25% 26%-50% 51%-99% 100%

22 respondents

21 respondents

Survey Survey RespondentsRespondents

9

Results, part 1:Results, part 1:Major QuestionsMajor Questions

10

Overall SuccessOverall Success“How successful do you feel the KCFTC has been in “How successful do you feel the KCFTC has been in

accomplishing its goals overall?”accomplishing its goals overall?”

0%

10%

20%

30%

40%

50%

All Staff 3% 3% 47% 35% 9%

Not at all successful

A little bit successful

Somewhat Successful

Moderately sucessful

Extremely successful

11

Overall Success Overall Success (n=33)(n=33)“How successful do you feel the KCFTC has been in “How successful do you feel the KCFTC has been in

accomplishing its goals overall?”accomplishing its goals overall?”

0%

10%

20%

30%

40%

50%

60%

70%

80%

Time <25% 0% 6% 29% 53% 12%

Time >25% 6% 0% 69% 19% 6%

Not at all successful

A little bit successful

Somewhat successful

Moderately sucessful

Extremely successful

Time <25%: Mean 3.71 Std Dev. .772 Time >25%: Mean 3.19 Std. Dev. .834

Between Group Comparison: Front-line (>25% with Between Group Comparison: Front-line (>25% with KCFTC) versus Other (<25%) Team MembersKCFTC) versus Other (<25%) Team Members

12

Overall Success Overall Success (compared to (compared to regular court)regular court)

“Compared to regular dependency court process, how “Compared to regular dependency court process, how successful do you feel the KCFTC is in accomplishing successful do you feel the KCFTC is in accomplishing its overall goals for participating families?” (n=33)its overall goals for participating families?” (n=33)

0%

5%

10%

15%

20%

25%

30%

All Staff 6% 6% 21% 27% 27% 12%

Somewhat less

About the same

A little bit more

Somewhat more

A good deal more

Much more succesful

13

Overall Success Overall Success (compared to (compared to regular court)regular court)

“Compared to regular dependency court process, how “Compared to regular dependency court process, how successful do you feel the KCFTC is in accomplishing successful do you feel the KCFTC is in accomplishing its overall goals for participating families?” (n=33) its overall goals for participating families?” (n=33)

0%

10%

20%

30%

40%

50%

60%

Time <25% 6% 6% 12% 35% 29% 12%

Time >25% 6% 6% 29% 53% 12%

Somewhat less

About the same

A little bit more

Somewhat more

A good deal more

Much more successful

Time <25%: Mean 5.12 Std Dev. 1.317 Time >25%: Mean 4.73 Std. Dev. 1.335

Between Group Between Group ComparisonComparison

14

Success Serving Target Success Serving Target PopulationPopulation

“How successful do you believe the KCFTC has been in “How successful do you believe the KCFTC has been in serving its target population?” (n=33)serving its target population?” (n=33)

0%5%

10%15%20%

25%30%35%40%45%

All Staff 3% 12% 36% 39% 9%

Not at all successful

A little bit successful

Somewhat Successful

Moderately sucessful

Extremely successful

15

Success Serving Target Success Serving Target PopulationPopulation

“How successful do you believe the KCFTC has been in “How successful do you believe the KCFTC has been in serving its target population?” (n=32)serving its target population?” (n=32)

0%

10%

20%

30%

40%

50%

60%

Time <25% 0% 6% 31% 50% 12%

Time >25% 6% 13% 44% 31% 6%

Not at all successful

A little bit successful

Somewhat successful

Moderately sucessful

Extremely successful

Time <25%: Mean 3.69 Std Dev. .793 Time >25%: Mean 3.19 Std. Dev. .981

Between Group ComparisonBetween Group Comparison

16

Success Serving Representative Success Serving Representative PopulationPopulation

“How would you describe the KCFTC’s success in serving “How would you describe the KCFTC’s success in serving a population that reflects the race, ethnicity, and gender a population that reflects the race, ethnicity, and gender of the general population of parents involved in the DCFS of the general population of parents involved in the DCFS

dependency system?” (n=30)dependency system?” (n=30)

0%

5%

10%

15%

20%

25%

30%

35%

All Staff 23% 23% 30% 17% 7%

Not at all successful

A little bit successful

Somewhat Successful

Moderately sucessful

Extremely successful

17

Success Serving Representative Success Serving Representative PopulationPopulation

“How would you describe the KCFTC’s success in serving a “How would you describe the KCFTC’s success in serving a population that reflects the race, ethnicity, and gender of population that reflects the race, ethnicity, and gender of the general population of parents involved in the DCFS the general population of parents involved in the DCFS

dependency system?” (n=29)dependency system?” (n=29)

0%

5%

10%

15%

20%

25%

30%

35%

40%

Time <25% 36% 21% 36% 7% 0%

Time >25% 13% 20% 27% 27% 13%

Not at all successful

A little bit successful

Somewhat successful

Moderately sucessful

Extremely successful

Time <25%: Mean 2.14 Std Dev. 1.027 Time >25%: Mean 3.07 Std. Dev. 1.280

Between Group ComparisonBetween Group Comparison

18

Summary of Major Summary of Major QuestionsQuestions

00.5

11.52

2.53

3.54

<25% 3.71 3.69 2.14

Front line 3.19 3.19 3.07

All respondents 3.45 3.39 2.6

Overall SuccessServing target population

Serving representative population

19

KCFTC CapacityKCFTC Capacity“In your opinion is the KCFTC currently serving too “In your opinion is the KCFTC currently serving too many families for its capacity, too few, or just about many families for its capacity, too few, or just about

right?” (n=32)right?” (n=32)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

All Staff 13% 81% 6%

Too Few Just About Right Too Many

20

Success of Referral and Success of Referral and Eligibility ProcessEligibility Process

“Overall, how successful do you think the KCFTC “Overall, how successful do you think the KCFTC referral and eligibility process has been?” (n=30)referral and eligibility process has been?” (n=30)

0%

5%

10%

15%

20%

25%

30%

35%

All Staff 7% 30% 29% 26% 3% 11%

Not at all successful

A little bit successful

Somewhat Successful

Moderately sucessful

Extremely successful

Don't Know

21

FTC CollaborationFTC Collaboration“A primary component of the KCFTC is “non-adversarial “A primary component of the KCFTC is “non-adversarial teamwork, in which team members know each other as teamwork, in which team members know each other as

individuals.” How successful have the KCFTC team members individuals.” How successful have the KCFTC team members and agencies been in achieving this ideal?” (n=34)and agencies been in achieving this ideal?” (n=34)

0%

5%

10%

15%

20%

25%

30%

35%

40%

All Staff 0% 15% 35% 38% 12%

Not at all successful

A little bit successful

Somewhat Successful

Moderately sucessful

Extremely successful

22

FTC Shared VisionFTC Shared Vision“Please rate the extent to which there is a shared vision “Please rate the extent to which there is a shared vision

among the different FTC team members. (n=34)among the different FTC team members. (n=34)

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

All Staff 15% 27% 41% 18% 12%

No shared vision

A little bit of a shared vision

Some shared vision

A good amount of shared vision

Substantial shared vision

23

Results, part 2:Results, part 2:KCFTC Processes KCFTC Processes

and Functionsand Functions

24

3.55

3.82

3.09

3.39

4.13

3.03

3.88

3.61

3.77

3.09

3.29

3

2 2.5 3 3.5 4 4.5 5

Comprehensive SB Assessment

High quality, appropriate CD Services

Timely/effective other services (parents)

Timely/effective other services (children)

Care planning and management

Expanded and more freq visitation

Consistent, timely incentives & sanctions

Random UA Screens

Effective pre-hearing case conferences

Training and education for FTC staff

Effective judicial interaction

FTC staff collab with other agencies

Summary of Process and Function Questions

A little bit successful

Moderately

Somewhat

Extremely

25

Results, part 3:Results, part 3:Achievement of Achievement of Best PracticesBest Practices

26

4.75

4.7

5.1

4.94

3.75

4.79

5.71

4.23

4.67

5.47

5.48

5.53

3 3.5 4 4.5 5 5.5 6

Court+providers maintain communication

Judge plays active role in Tx process

Judge responds to positive & neg behavior

Mechanisms for shared decision making

Accountability for Tx services

AOD testing frequent in first months

Strategy for responding to noncompliance

MIS allow data to be assembled/reviewed

Children protected from abuse and neglect

Children kept safely in their homes possible

Enhancement of due process

Team members provided adeq resources

Summary of Best Practices Questions

About the same Somewhat more

Little more successful

Good deal more

27

Results, part 4:Results, part 4:Proposed KCFTC Proposed KCFTC

OutcomesOutcomes

28

4.73

4.87

4.97

5.19

5.09

4.97

4.9

4.7

5.36

4.65

5.13

5.3

3 3.5 4 4.5 5 5.5 6

Eligibility/enrollment completed quickly

Enrollment in appropriate CD services

Parents compliant with/complete Tx

Ultimately able to be and remain sober

Parents/childr more fully engaged in svcs

Parents/ch receive services they need

Decreased plcement disruptions

Parents compliant with court orders

Less negative effect on child well-being

Less disruption of child-parent bonds

Increased family reunification rates

Earlier determ. of alternate plcmt options

Summary of Outcomes Questions

About the same Somewhat more

Little more successful

Good deal more

29

Results, part 5:Results, part 5:Open ended Open ended questionsquestions

30

Gaps in Resources, Barriers, Gaps in Resources, Barriers, and other issues that need to and other issues that need to

be addressed.be addressed.THEME Responses

Percent of Total (n=231)

Client-Related Needs 39 18% Resources and Services 28 12%

Housing 12  

Visitations 10  

Transportation 4  

Day Care 2  

Focus on the Child 3  

Selection 2  

Outreach 2  

Other 2  

Support 1  

Outcomes 1  

Treatment Issues 38 16% Choice of Providers 16  

Other/Unspecified 14  

Wraparound 4  

Inpatient Treatment 2  

Strength-Based Approach 2  

31

THEME ResponsesPercent of Total

(n=231)

Communication and Collaboration 29 13% Team Building 17  

Staffings 5  

Wraparound 4  

Uniformity 2  

Other 1  

Funding Needs 27 12% General Funding Needs 19  

Expansion 3  

Incentives for Clients/Families 3  

Donations 2  

Training Needs 27 12% No Time or Money for Training 12  

About the Program Specifically 10  

To Facilitate Cooperation 5  

FTC Procedures 22 9% Established Policy 11  

Organization 6  

Other 5  

Gaps continued…

32

THEME ResponsesPercent of Total

(n=231)

Technology 18 8%

MIS 12  

Computers 6  

Administration 12 5%

Support to the Team 6  

General 6  

Time 8 3%

Better Use of Time 3  

Time Constraints 5  

No Response/Don't Know 11 5%

Gaps continued…

33

THEME ResponsesPercent of Total

n=137

KCFTC Model/Program Strengths 38 28% Model Itself Facilitates Success 20 15%

Strength-Based Approach and Vision 9  

General 6  

Wraparound 5  

Day Care 2  

Scope and Intensity of the Program 10  

Accountability for Everyone 8 6%

Low Case Loads 5

General 3  

Team Members 31 23% Investment/Involvement/Commitment/Dedication 18  

Quality of Providers and Treatments 13  

General Strengths of the KCFTC

34

THEME ResponsesPercent of Total

n=137

Court 23 17% Frequency of Hearings/Proceedings 10  

Court Atmosphere - Positive 7  

Accessibility 5  

Presence of Team Member 1  

Collaboration 18 13% Between Team Members 12  

In Decision-Making 4  

Team Members and Clients 2  

Participation 17 12% General Participation of Families in FTC 12  

Establishes Relationship 4  

Promotes Reunification 1  

Treatment Providers 10 7% Treatment Providers in General 5  

Individualized/Tailored Treatment 5  

Strengths continued…

35

Greatest Challenges or Greatest Challenges or Weaknesses of the Weaknesses of the

KCFTCKCFTCTHEME Responses

Percent of Total (n=112)

Team Members Issues 41 37%

Team Concept 14 13%

Team Roles 9  

Team Stability 3  

Team Member Involvement 2  

Cohesiveness 11  

Communication 10 9%

Between Team Members 9  

With Treatment Providers 1  

Organization/Management 2  

Education and Training 2  

Outcomes 2  

36

THEME ResponsesPercent of Total

(n=112)

Treatment 24 21% Choice of Providers 10 9%

General Choice 8  

Individualized 2  

Quality of Treatment 5  

Inpatient Treatment 3  

Unclassified Remarks 2  

Instability of Current Treatment 1  

Equality for All Clients 1  

Timeliness of Service Not Good 1  

Approach/Vision of Program 17 15% Clarity of Vision/Mission of Program 7  

Focus Needs to be on Child 6  

Strength-Based 4  

Challenges continued…

37

THEME ResponsesPercent of Total

(n=112)

Client's/Families 12 11%

Selection/Referral Process 5  

Level of Engagement 4  

Educating Clients 3  

Visitations 1  

Court 9 8%

Rewards and Sanctions 6  

Court in General Needs Work 3  

Accountability 5 5%

Financial Considerations 4 4%

Challenges continued…

38

Additional Results:Additional Results:Ratings of Ratings of

Individual Team Individual Team Members’ Members’

EffectivenessEffectivenessEffectiveness at being a non-Effectiveness at being a non-adversarial team memberadversarial team member

Overall effectiveness at achieving Overall effectiveness at achieving positive outcomes for enrolled positive outcomes for enrolled familiesfamilies

39

JudgeJudgeNon-Adversarial Team Member Overall Non-Adversarial Team Member Overall

Effectiveness (n=19)Effectiveness (n=19)

0%

10%

20%

30%

40%

50%

60%

70%

All Staff 0% 0% 5% 37% 58%

No at all successful

A little bit successful

Somewhat successful

Moderately successful

Extremely successful

0%

10%

20%

30%

40%

50%

60%

70%

All Staff 0% 0% 21% 58% 21%

Not at all successful

A little bit successful

Somewhat successful

Moderately successful

Extremely successful

40

DSHS Social WorkerDSHS Social WorkerNon-Adversarial Team Member Overall Non-Adversarial Team Member Overall

Effectiveness (n=19)Effectiveness (n=19)

0%

10%

20%

30%

40%

50%

60%

70%

All Staff 0% 5% 0% 37% 58%

No at all successful

A little bit successful

Somewhat successful

Moderately successful

Extremely successful

0%5%

10%15%20%25%30%35%40%45%50%

All Staff 0% 0% 11% 47% 42%

Not at all successful

A little bit successful

Somewhat successful

Moderately successful

Extremely successful

41

Team members ratedTeam members rated

JudgeJudge DSHS Social DSHS Social

WorkersWorkers Social Work Social Work

SupervisorSupervisor CASACASA CASA CASA

Supervisor/MgrSupervisor/Mgr KCFTC Program MgrKCFTC Program Mgr KCFTC SpecialistKCFTC Specialist

PCAP PCAP RepresentativesRepresentatives

Treatment providerTreatment provider Tx provider Tx provider

supervisorsupervisor Child’s attorneyChild’s attorney Parents’ attorneyParents’ attorney Wraparound Wraparound

facilitatorfacilitator

42

SUMMARY OF RESULTSSUMMARY OF RESULTS 88% believe the KCFTC is at least “a little 88% believe the KCFTC is at least “a little

bit more successful” than the regular bit more successful” than the regular dependency court at achieving outcomesdependency court at achieving outcomes Overall, 94% of respondents believe the KCFTC Overall, 94% of respondents believe the KCFTC

has been at least “somewhat successful” overallhas been at least “somewhat successful” overall Advisors and those with less overall contact with Advisors and those with less overall contact with

KCFTC participants gave higher ratingsKCFTC participants gave higher ratings Respondents less confident that goal of Respondents less confident that goal of

serving a representative population is being serving a representative population is being metmet

Respondents overwhelmingly believe Respondents overwhelmingly believe current client load is about right for capacitycurrent client load is about right for capacity

Mixed opinions on the amount of shared Mixed opinions on the amount of shared vision among team membersvision among team members

43

Relative strengthsRelative strengths

Processes, functions, and best practicesProcesses, functions, and best practices VisitationVisitation Random UA screens and AOD testingRandom UA screens and AOD testing Judicial interactionJudicial interaction Communication between court and providersCommunication between court and providers

Outcomes (compared to regular Outcomes (compared to regular dependency court)dependency court) Parents’ ability to remain soberParents’ ability to remain sober Reduction of negative effects on childReduction of negative effects on child Parents’ compliance with court ordersParents’ compliance with court orders Enrollment in appropriate CD servicesEnrollment in appropriate CD services

44

Relative weaknessesRelative weaknesses Processes, functions, and best practicesProcesses, functions, and best practices

Strengths-based assessmentsStrengths-based assessments Timely and effective MH and other services for Timely and effective MH and other services for

parentsparents Consistent and timely sanctions and incentivesConsistent and timely sanctions and incentives Staff training and educationStaff training and education MISMIS CD Treatment accountabilityCD Treatment accountability

Outcomes (compared to regular dependency Outcomes (compared to regular dependency court)court) Compliance with and completion of treatmentCompliance with and completion of treatment Full engagement in servicesFull engagement in services Earlier determination of alternative placement optionsEarlier determination of alternative placement options

45

Areas that need Areas that need improvementimprovement

(From open-ended questions)(From open-ended questions)• Resources and Services for clients

• Housing• Visitations• Transportation• Day Care

• Treatment services• Greater individualization and choice for services

• Improved Communication and Collaboration• Team Building• Staffings

• Increased Funding and Sustainability• Training Needs – Especially to ensure understanding of the model

and to facilitate shared vision

• More fully established policies and procedures• MIS and data

46

Next stepsNext steps

Complete surveying and analysisComplete surveying and analysis Complete analysis of Open-ended Complete analysis of Open-ended

questionsquestions Develop reports and presentations Develop reports and presentations

based on needsbased on needs Move ahead with Participant-level Move ahead with Participant-level

component of evaluationcomponent of evaluation