performance assessments can: help identify potential problems in the program help identify areas...

Why should your drug court care about performance improvement processes?

Presented by: David Wright, Ph.D. & Lorrie Byrum

You’re Good &

You Know It!

Performance assessments can: help identify potential problems in the program help identify areas where streamlining the process could be useful & help make the program run more smoothly

help ensure fidelity to the model help ensure that the program is following the 10 key components/10 essential elements

The Importance of Performance Assessment Within Your Drug Court

Why should we care if we adhere to the 10 key components:

Courts that bring people into the program within 50 days of arrest have greater impact on reducing recidivism

Courts that drug test at least two times per week (with a true random pattern) are twice as cost effective

Courts that use jail sanctions no longer than 6 days had lower recidivism than courts with jail sanctions of 6+ days

DRUG COURTS THAT EVALUATE THE PROGRAM & MODIFY THE OPERATION BASED ON RESULTS HAD TWICE THE COST SAVINGS OF OTHER COURTS

The Importance of Performance Assessment Within Your Drug Court

How Does My Court Compare To The 10 Key Components

Taking The Temperature Of The Court

How do you evaluate your program like component

number 8 indicates?

Key Component #8:“Monitoring and evaluation measure the achievement of program goals and

gauge effectiveness.”

WEBS Is Your Data Collection Process Online

Reports

Active Counts

Outcomes

Performance Measures

Admission Data

Referral Data (FDC & MHC Only)

Phase Data

Discharge Data (Termination,

Graduation, etc.)

http://www.themagazine.ca/wp-content/uploads/2009/01/GarbageTruck.jpg

Importance of Accurate Data

Garbage In

Garbage Out

Source: http://www.trashitman.com/blog/wp-content/uploads/2011/03/Packer-truck.jpg

Inaccurate Data Affects…

•Your Court’s Active Counts

•Your Court’s Outcomes & Performance Measures

•& Ultimately, Statewide Data that Affects All Specialty Courts

Performance Improvement

Ensure Current DataData Accuracy Goal

Setting

Case Review

Identify Systemic Issues

Ensure

Contract

Compliance

Monitor

Performance

Improvement

Ensure Needs Are Being Met

Performance Improvement

Ensuring data quality:Contractual requirementTied to fundingData matchingData Review

Contracts require data elements for all active, graduated, & terminated participants.

Participants are only counted active for purposes of funding if data records have been updated in at least the last 60 days.

Additionally, contract payments can be withheld for failure to accurately report data.

The ODMHSAS holds data sharing agreements with multiple agencies including the Oklahoma Department of Corrections & the Oklahoma Employment Securities Commission.

Run program data & review for outliers or data inconsistencies, correct errors.

Performance ImprovementTied to Funding

When a review of the program’s 60 Day Active Count shows a participant or a group of participants that have not been updated in the last 60 days program staff should take steps to ensure the data are updated immediately or by the 2nd of each month. This ensures programs have current data.

Performance ImprovementData Matching

Performance ImprovementData Matching

Performance ImprovementData Matching

Performance ImprovementData Matching

Performance ImprovementData Review

When a review of outcome data shows a dramatic change, courts should review data to determine if outliers. Once identified, then correct inaccuracies. This ensures programs have accurate data.

Performance ImprovementData Review

1) Run participant-level reports to identify the outliers (Example: Which participants have decreasing income between admission & graduation.)

2) Verify data through file review (Example 1: Pull participant file & review employment verification/pay stub to verify income at graduation.)

3) Correct record to reflect the accurate income.

4) Re-run report to verify correction & view new average for this outcome at the court level (correcting individual cases impacts the overall court results).

Performance ImprovementCase Studies

When a review of outcome data shows an outlier, courts can verify & correct data (as seen previously). IF data are correct, then courts can conduct case studies to determine the explanation for why the case was an outlier.

Performance ImprovementCase Studies

1) Identify the outlier of focus

2) Case ‘Wwwww’ has no prior felony convictions, no prior arrests, no additional charges; & the current charge is Possession for which they received a 10 year prison sentence. Why would this case be in your court? The reasons are obvious for ‘meth user’, ‘Tt Tt’, & ‘nancy warren’

3) Begin case study gathering information on why this person came into the program. Review:

• Risk Screen Information (participant has strong association with known criminals, is unemployed, is homeless, & has no family support)

• ASI (participant has used drugs daily for the past15 years & began at an early age)

• Mental Health Screen (participant has a co-occurring mental health concern).

Performance ImprovementGoal Setting

With the ability to run real-time outcome reports, courts can set goals for improvement of one or more outcome measures.

Performance Improvement

Goal Setting1) Identify which outcome(s) to target

2) Set a measureable, realistic goal for improvement

3) Set a timeline to re-evaluate

Performance Improvement

Goal Setting1) Identify which outcome(s) to target

2) Set a measureable, realistic goal for improvement

3) Set a timeline to re-evaluate

4) Identify a barrier/barriers which has lead to the current issue

• Many times this is simply, it wasn’t a priority. • (Case Studies vs. Systemic Issues)

5) Choose one barrier & identify steps to remove/reduce it.

6) Re-evaluate

Performance ImprovementSystemic Issues

Data can reflect a systemic issue that needs to be addressed through policy change(s).

The use of jail as a sanction in MHC is:• Ineffective (costly and does not result in positive

behaviors)• Inappropriate• AGAINST best practices• Harmful

Performance ImprovementSystemic Issues

Data can reflect a systemic issue that needs to be addressed through policy change(s).

NOT getting people into treatment quickly is:• AGAINST best practices• Harmful to recovery (results in poor outcomes)• Delays needed treatment

Performance ImprovementSystemic Issues

1) Identify which outcome(s) to target

2) Run participant-level reports which show multiple participants impacting an outcome measure.

3) Meet with the court team to develop plan of action. Multiple participants are impacted. (Example 1: Extended jail days being used as a routine sanction. May need to re-evaluate sanction matrix. Example 2: Excessive time between referral & admission. May need to review the admissions process.)

4) Identify a timeline to re-evaluate the plan from step 3.

5) Re-evaluate

Performance Improvement Goal Setting or Systemic Issues

Performance ImprovementGoal Setting or Systemic Issues

1) Run demographic reports which show the make-up of the participants within the court.

2) Compare the court participant demographics to that of arrests within the county and the county demographics as a whole. Does the program mirror these demographics?

3) Meet with the court team to determine if the issue is due to goal setting or if the issue is systemic then develop plan of action to address the issue.

4) Identify a timeline to re-evaluate the plan from step 3.

5) Re-evaluate

Court Gender Comparisons

Male Female0%

25%

50%

75%

100%

49.5% 50.5%

75.5%

24.5%

86.1%

13.9%

64.5%

35.5%

71.4%

28.6%

Oklahoma Population All Arrests Prison Receptions

Mental Health Court Admissions Drug Court

Court Race/Ethnicity Comparisons

0%

20%

40%

60%

80%75%

8% 9% 10%

73%

17%

9%6%

56%

25%

11%

7%

69%

17%12%

1%

69%

9%

19%

3%

Oklahoma Population All Arrests Prison Receptions

Mental Health Court Admissions Drug Court

Performance Improvement

State representative role (Field Representative)

Building a relationship with the programsKeeping data as a priorityCommunicating with the programs on outcome changes

Identifying training needs/trendsAssisting in the assurance of quality dataProviding technical assistance (case studies & systemic concerns)

Reports Available on Demand

Other Uses of the Reports

Criminal Info Report Check to see if reaching target population Ensure accuracy of data used to calculate

your court’s priors data

Outcome Page Report Use to check other performance & outcomes

measures

Demographics ReportsEnsure your court is reaching its target

population

Sharing Program Successes with Community Stakeholders

Completion Rates by Race

Caucasian African American

Native American

Hispanic Total52%

54%

56%

58%

60%

62%

64%

66%

68%

70%

72%

69.0%

59.0%

62.0%

70.0%

66.0%

Unemployment by Race

Caucasian African American

Native American

Hispanic Total0%

5%

10%

15%

20%

25%

30%

35%

40%

34.0%

31.0%

38.0%

21.0%

34.0%

3.0% 4.0% 3.0%2.0%

3.0%

Entry Graduation

Income from Employment by Race

Caucasian African American

Native American

Hispanic Total$0

$200

$400

$600

$800

$1,000

$1,200

$1,400

$1,600

$1,800

$870

$577 $613

$1,046

$799

$1,700

$1,052

$1,406

$1,559 $1,574

Entry Graduation

Percent without H.S. Diploma/GED

by Race

Caucasian African American

Native American

Hispanic Total0%

10%

20%

30%

40%

50%

60%

70%

25.0% 25.0%27.0%

58.0%

26.0%

16.0%18.0%

16.0%

40.0%

17.0%

Entry Graduation

Percent of Children Living with Participants by Race

Caucasian African American

Native American

Hispanic Total0%

10%

20%

30%

40%

50%

60%

70%

39.0% 37.0% 41.0%44.0%

40.0%

60.0% 59.0%63.0% 61.0% 60.0%

Entry Graduation

Conclusion

Run all reports regularly Ensure data are current Ensure data accuracy Confirm contract compliance Monitor improvement Review case abnormalities Set goals Address systemic issues Serve county needs effectively• Risk, MH, SA, Race/Ethnicity, Gender

Contact Info

David Wright (405) 522-6169 dwright@odmhsas.org

Lorrie Byrum (405) 522-6172 lbyrum@odmhsas.org

Nancy Warren (405) 522-6170 nwarren@odmhsas.org