tracking participant progress during covid-19...tracking participant progress during covid-19 april...

W H A T T O C O L L E C T A N D W H Y

T R A C K I N G PA R T I C I PA N T P R O G R E S S D U R I N G C O V I D - 1 9

A P R I L 2 0 2 0

3

DUE TO THE COVID-19 PANDEMIC, WE HAVE HAD TO CHANGE HOW OUR TREATMENT COURTS OPERATE

Are these changes harming our participants?

Will the changes in supervision and monitoring threaten public safety?

How do we help our participants continue to make progress?

What are they up to?

4

THE FACTOR CAUSING THE MOST ANXIETY IS THE UNKNOWN

5

GOOD NEWS!

Collecting data, and measuring outcomes, can make the unknown, known.

6

C R I S I S

A crisis is a terrible thing to waste

7

POLL: DANGER - WHAT HAVE YOU CHANGED THAT YOU ARE MOST CONCERNED WILL HARM PARTICIPANTS?

• Virtual court sessions• Virtual treatment• Virtual support groups/peer

support• Virtual/phone supervision• New drug test method(s)• Stopped drug testing• Stopped jail sanctions

8

POLL: OPPORTUNITY - WHAT HAVE YOU CHANGED THAT YOU THINK IS MOST BENEFICIAL FOR YOUR PARTICIPANTS?

• Virtual court sessions• Virtual treatment• Virtual support groups/peer

support• Virtual/phone supervision• New drug test method(s)• Stopped drug testing• Stopped jail sanctions

9

1 WHY DO WE COLLECT DATA?

WHAT DO WE COLLECT?

HOW DO WE USE OUR DATA TO FIND OUT IF WHAT WE’RE DOING IS WORKING?

OVERVIEW

2

3

4 WHAT TOOLS ARE AVAILABLE FOR COLLECTING AND MONITORING DATA?

5 QUESTIONS

10

10

W H Y D O W E C O L L E C T D ATA ?

Three Main Reasons

11

11

So we can track what our participants are doing, and respond

appropriately to their behavior

WHY DO WE COLLECT DATA?

12

12

So we can see if what we’re doing is working


13

13


So we can improve what we’re doing

14

14

20% 20%

41%37%

Review Data(n=38), p<.05

Used Evaluation(n=34), p<.10

YesYesNo No

105%85%

Carey, Finigan, & Pukstas (2008);Carey, Mackin, & Finigan (2012)

REDUCTIONS IN RECIDIVISM

15

15

MORE GOOD NEWS!

The data you need to track participant progress is the same data you need to find out if what you are doing is working and the same data you need to find out how to improve your performance

16

16

W H AT D O W E C O L L E C T ?

What you are doing (and any changes)

What participants are doing

17

FIRST: TRACK WHAT YOU’RE DOING

• Track major policy and practice changes (Write them down)

18

FIRST: TRACK WHAT YOU’RE DOING

• Key data: Include policy change and dates for when you made those changes

o The dates allow you to know which participants experienced those new practices and policies

o Then you can look at outcomes for participants before those changes were made and after, and see which group did better

• Track major policy and practice changes (Write them down)

19

SAMPLE TIMELINE OF CHANGES IN POLICY AND PRACTICE

2012

2013

2014

Reduced the number of defense attorney(s) to a dedicated pool, eventuallyresulting in a single dedicated defense attorney (in late 2013)

Required treatment providers and case managers to be represented duringcourt sessionsImplemented random drug testing

Implemented regular pre-court staffing meetings with entire team presentAdded law enforcement representative to the Steering Committee

Implemented new incentives and sanctions guidelines including; limiting jailsanctions, increasing variety of both incentives and sanctions, increasingfocus on rewards for positive behaviors, and tracking responses in programdata system

Judge rotation (Judge Blank)

Increased the frequency of status hearings to twice per month in the firsttwo phases

Added a designated role for a drug court coordinator

Aug

Aug

Jan

Oct

Mar

20

WHAT HAVE YOU CHANGED? AND WHEN?

• Did your courthouse close?

• Did you change to virtual court sessions (video or phone/conference calls)? Did you stop court sessions?

• Did you purchase cell phones or tablets for participants who don’t have them?

• Did team members start calling participants to check on how they’re doing and to perform remote case management?

• Did you start virtual treatment sessions?

21


Policy Change DateMM/YY

MM/YY

MM/YY

MM/YY

MM/YY

• Did your courthouse close?

• Did you change to virtual court sessions (video or phone/conference calls)? Did you stop court sessions?

• Did you purchase cell phones or tablets for participants who don’t have them?

• Did team members start calling participants to check on how they’re doing and to perform remote case management?

• Did you start virtual treatment sessions?

22


• Did you start virtual peer support meetings?

• Did you change the type of drug testing? Did you stop drug testing? Did you change the drug testing protocol (e.g., unobserved testing)?

• Did you introduce some new incentives?

• Did you decrease sanctions? Did you stop using jail?

• Did your requirements for participants change and were those new requirements communicated to participants?

23

• Information about the activities you require of participants (many of these are what YOU are doing)

• Participant attendance at those activities

• These are the same kinds of data points you would collect regardless of the pandemic!

NEXT : COLLECT WHAT YOUR PARTICIPANTS ARE DOING

24

TYPICAL ACTIVITIES TO TRACK

Court Hearings Drug Tests Incentives Sanctions Treatment Case management Supervision Peer Support What else do you do?

25

S P E C I F I C D ATA E L E M E N T S T O

C O L L E C T

Individual Level(for each participant)

26

PARTICIPANT IDENTIFIERS AND BACKGROUND CHARACTERISTICS

Name and other identifiers:o SSNo Drivers license numbero Court case numbers

Demographicso DOBo Raceo Gender

Referral Information Risk and Need Assessments

(dates and results)

Other characteristics at entry

o Employment statuso Incomeo Housing statuso Child custody/visitationo Marital statuso Other social indicators

27

SOCIAL INDICATORS OVER TIME

Collected at multiple timepoints: At program entryDuring programAt program exit

o Employment statuso Incomeo Housing statuso Child custody/visitationo Marital statuso Other social indicators

28

Dates of HearingsVirtual/In-PersonAttendanceOther Descriptors:

o Judge presidingo Phase promotiono Incentives/Sanctions

COURT HEARINGS

Nec essary Informati on:

29

DRUG TESTS

Dates of EACH Drug TestDrug Test Type:

o UA, EtG, PBT, oral swab, bracelet, patch, hair, blood, ignition interlock, etc.

Observation MethodSubstances Tested Results of Drug Tests:

o Negativeo Positive (including refused,

diluted, or no-show)


30

Behavior o 100% attendance, Missed UADates of ResponsesResponseo Gift card, Writing assignmentResponse Typeo Incentive, Sanction, Therapeutic,

Supervision AdjustmentDuration (if applicable)o Jail Days (Entry/Exit Dates or #

of Days), Community Service Hours

INCENTIVES ANDSANCTIONS


31

TREATMENT

Dates of Treatment Sessions (or Episode Start & End Dates) Virtual/In-PersonDuration of Treatment

o Could be days for inpatient/residential

o Could be hours for outpatient appointments

Type or modalityCompletion/Discharge

StatusAgency Providing Treatment


32

TREATMENT

Inpatient/ResidentialDetoxTransitional Individual OutpatientGroup OutpatientMedication Assisted

Therapy (MAT)Assessment, Case

Management Activities

Treatment Modal i t i es:

33

PROGRAM STATUS

Entry DatePhase Start Dates(for each phase)Exit DateStatus at Exit

o Graduatedo Terminatedo Transferred


34

RECIDIVISM

Dates of arrestCharge type/descriptionFelony/Misdemeanor

New Arrests and Charg es:

35

Dates of activityType of activityAttendanceVirtual/In-Person

OTHER ACTIVITIES


Peer support groups, case management, supervision

36

HOW DO WE TRACK WHAT PARTICIPANTS ARE DOING?

Regular phone check-ins (record dates of check-ins and whether participant picked up)

Do home visits and have them come out in front to talk (from a proper 6’ distance) (record dates of home visits and whether they were home)

Do you and your participants have access to technology that allow you to stay in touch with each other (phones/computers/tablets?)?

37

SELF-REPORT (DATA GATHERED FROM PARTICIPANTS)

Indicators of progress:o Do they have a safe place to live? o Do they have an income? o Do they have a way to get food? o How has the pandemic impacted their employment? o Do they have children at home? Do they have what they

need to help their kids connect with school? o Are they attending peer support groups? o Are they connecting with peer specialists? o Are they attending (online) treatment sessions?

Record the answers to these questions with a date (at entry, during, and at exit)

*Some of this information should also be gathered from other sources for confirmation

38

H O W D O W E U S E O U R D ATA T O F I N D O U T I F W H AT W E ’ R E

D O I N G I S W O R K I N G ?

Participant & Program Performance Measures

39

• What percentage of participants attend treatment consistently? What is the average number of treatment sessions attended/missed per week?

• What percentage of participants attend case manager (CM) appointments consistently? What is the average number of treatment CM appt’s that have been missed? Is that number different for different CM’s?

• What percentage of participants attend peer support meetings consistently? What meetings are attended most frequently?

• What is the percent of positive drug tests?

SHORT TERM PROGRESS INDICATORS ( W E E K S)

For i ndi vi dual part i c i pants and prog ram performanc e:

40

During program:• What percent of participants are employed or in school?• What percent of participants have safe and sober housing?• What percent of participants drop out/are terminated in

Phase 1, or 2, or…X?• Do average risk and need scores decrease over time?• What is your participant graduation/commencement rate?

During and after program exit:• What percent of participants get rearrested during the

program? In the first year after exit? In the second year after exit?

MID AND LONG-TERM PROGRESS INDICATORS ( M O N T H S TO Y E A R S)

Mostly prog ram performanc e:

41

HOW TO EXAMINE THE INDIVIDUALPROGRESS INDICATORS OVER TIME

15%

7%

2%0% 0%

Phase 1 Phase 2 Phase 3 Phase 4 Phase 5

Percent of Positive Drug Screens: Shannon Carey

Employment

42

HOW TO EXAMINE YOUR PROGRAMPROGRESS INDICATORS OVER TIME

Calculate the attendance rates (percent of events attended)Calculate the average number of events attended/missed

per time period (e.g., week/month) Compare the rates before and after policy changes

70%

90%

Jan - Feb 2020 Mar - Apr 2020

Outpatient Treatment Attendance Rate

Started virtual treatment sessions for

all participants:March 2020

43

W H AT T O O L S A R E A V A I L A B L E F O R

C O L L E C T I N G A N D M O N I T O R I N G

D ATA ?

44

POLL: WHAT DATA SYSTEM DO YOU HAVE TO TRACK YOUR PARTICIPANTS?

• Statewide web-based system• Individual court web-based

system• Access or other non-web-

based database (resides on your individual computer)

• Spreadsheet (Excel) on a computer

• Paper records

45

OPTIONS FOR GENERATING REPORTS

• If you have a database/case management system already, it may calculate the numbers for you

o Ask your IT department to create automated reports with those numbers

o If not, see if you can download the data into a spreadsheet

• If you don’t have a database, you can enter the data into a spreadsheet, like Excel and calculate some basic statistics

o Simple calculations (e.g., averages, sums)o Pivot Tables and Pivot Chartso Graphso Dashboards (advanced)

46

SPREADSHEET – Participant info

47

47

DASHBOARD EXAMPLE – Change in risk level

48

48

DASHBOARD EXAMPLE – Graduation Rates

49

49

DASHBOARD EXAMPLE - Recidivism

50

50

EXCEL RESOURCES

• Using drop-down lists in Excel:https://support.office.com/en-us/article/Create-a-drop-down-list-7693307a-59ef-400a-b769-c5402dce407b

• Using pivot tables in Excel:https://support.office.com/en-us/article/Create-a-PivotTable-to-analyze-worksheet-data-a9a84538-bfe9-40a9-a8e9-f99134456576

51

51

Q U E S T I O N S ?

Shannon [email protected]

Charlene [email protected]

T H A N K YO U

W W W. N P C R E S E A R C H . C O M

tracking participant progress during covid-19...tracking participant progress during covid-19 april...

Documents