Virginia Department for Aging and Rehabilitative Services

1

Barbara J. Burkett, Ph.D., M.S.P.H.Division of Policy and Planning 804-726-1911

Report Requested by the Commissioner and Field Services Director on the following: Utilization of vocational evaluation services Program’s effectiveness Outcomes Counselors’ satisfaction Best practices Other state’s use of vocational evaluation

2

3

Review of Internal DARS Data Number of Clients Served Timing of Vocational Evaluations Within The Rehabilitation Process Demographics of Clients Receiving Evaluations Outcomes for Clients Receiving Vocational Evaluations Compared to Those

Who Do Not Receive Vocational Evaluations Cost Metrics of Vocational Evaluations

DARS Case Audit Reviews Literature Search

Roles and Functions of Vocational Evaluators Review of Various State Vocational Rehabilitation Programs Use of Vocational

Evaluators Population Specific Instruments/Best Practices

Survey of DARS Counselors Regarding Current Program

Review of Internal DARS Data Data was extracted from AWARE, the DARS case

management system, and imported in to SPSS for statistical analysis.

Study Years: Federal Fiscal Years 2011, 2012 and 2013▪ Vocational Evaluation Cases Identified▪ 2011 case notes only▪ 2012 case notes and vendor file codes▪ 2013 case notes and vendor file codes

4

Case Notes String search for every iteration of “vocational evaluation” (Example: vocational evaluation, voc. eval., v.e., evaluation, evaluation, eval,….inclusive of v* and e* with and without

punctuation and capitalization) Verified string search results by opening each case in AWARE.

AND Vendor File

Career Services Comprehensive Evaluation Exploratory Evaluation Referral/Staffing/Consultation Report Writing

5

17 Vocational Evaluation Positions

15 Currently Filled 2 Vacancies (Eastern, Southwest)

6

Six percent of DARS clients received a vocational evaluation during the study period.

7

FFY

# Clients (Application –Post

Employment Status)

# Vocational Evaluations

% of All Clients In Application

through Post Employment Status

Receiving A Vocational Evaluation

2011 30,795 1,843 6.0%

2012 28,877 1,527 5.3%

2013 28,112 1,944 6.9%

There was no regional variation in requests for vocational evaluation.

There was great variation in the use of vocational evaluation by counselors.

The range in the number of vocational evaluations requested by DARS counselors during the three year time period was 1-127.

Average number of vocational evaluations requested was 8.8 per counselor per year (s.d of 8.7).

The mode was 6 requests per counselor per year.

8

9

Overall for the 3 year time period, transition students were more likely than non transition students to receive a vocational evaluation.

10

More males than females received vocational evaluations.

11

43% cognitive impairment 28.4% psychosocial impairment 7.9% other mental impairment

12

13

The rehabilitation rate was lower for clients who received a vocational evaluation over the 3 year period.

14

Average hourly earnings were lower for clients receiving a

vocational evaluation.

15

Clients who received a vocational evaluation were less likely to obtain a job at closure that was the same as indicated on the IPE.

Clients who received vocational evaluations were less likely to be referred for supported employment services.

16

17

The VR cost of the clients’ who received a vocational evaluation were lower than for those clients who did not receive a vocational evaluation.

Conducted by Scott Fraley and Tim OliveReport Written by Liz Smith

Selection Criteria for Cases Cases closed in FFY2013 that had a vocational

evaluation as a planned service Stratified random sample by region and caseload

type Total sample n=40

18

Five (12%) of the forty cases had a vocational evaluation scheduled, but never conducted. Two of these were caused by the consumer failing to appear for the vocational evaluation. The other three reasons were not evident from the case file.

35 Reports Twenty (57%) had no job recommendation in the vocational evaluation report.

Eight of the 20 cases with no recommendation (23%) were transition cases.

Of the fifteen cases that had a job recommendation in the vocational evaluation report: For twelve (80%) of the cases: the vocational evaluation recommendation, the

vocational goal in the IPE, and the job at case closure matched.

19

If it is the expectation that a job recommendation occur when a vocational evaluation is conducted, then that is not occurring on a consistent basis.

When a job recommendation is made in the vocational evaluation, there is a high level of consistency between that recommendation, the IPE goal, and the consumer’s job at case closure.

20

Response Rate For Survey: 85.7% n/N = 12/14

Number of Vocational Evaluations FFY13 x = 96.5 range = 76-123

Number of Vocational Evaluations April 2014 x = 10.6 Range 6-20 No show rate is a problem with 2 evaluators reporting a no show

rate of 46.4%

21

22

Referral Source DARS evaluators 1 report of parent/guardian 1 client request 1 supported employment vendor

Time From Referral to Evaluation X =41.7 days Range 20 – 65

Time to Complete Written Report X= 6.1 days Range = 2-16

Template Use 50% use now 25% would like to use 25% would not find it helpful

Job Recommendations Made 82% make job recommendation 18% did not (=2)

23

#1: Incomplete information on client at time of evaluation

#2: Client willingness to participate #3: Scheduling problems #4: Poor referrals #5: Client ability to participate

24

Discussion of vocational evaluation reports with DARS counselors 55% always discuss (n=6) 46% discuss sometimes (n=5) 1 person did not answer this question

25

Instruments Work Behavior Inventory used by 2 evaluators Threshold Monthly Work Evaluation: 100% not familiar with

instrument Vocational Cognitive Rating Scale: 100% not familiar with instrument Work Capacity Evaluation used by 2 evaluators Maryland Assessment of Social Competence: 100% not familiar Change Assessment for Vocational Counseling (URI): 100% not familiar IPS Fidelity Scale: 1 evaluator familiar with instrument Supported Employment Scale: 1 evaluator familiar with this instrument

26

Population Based Instruments

27

Use Do not use Would Like to Use

Ethnic Based 7 1 2

Minority Based

4 3 4

Gender Based 5 2 4

Hearing Impaired

5 1 5

Autism Based 8 0 3

Mental Illness 5 2 4

College Bound 9 0 2

Carolyn Graham, Ph.D. (VCU) contracted for literature review

Barbara Burkett, Ph.D. (DARS)

Best Practices: Guide to Vocational Assessment and Evaluation-4th Edition

Author: Paul Power “The State of Vocational Evaluators: A National Study” Sligar and Betters Journal of Rehabilitation Vol. 74, no. 4

28

Sligar and Betters interviewed 63 agencies

o 97% (n= 60) of states purchased vocational evaluation services from evaluators in private practice, psychologists, and community rehabilitation programs.

o Average number of vocational evaluators by agency was 15.1 with a range of 1-72.

o 65% of the respondents did not require a certificationo 81% of the programs had guidelines for the vocational evaluation and the

final report.

29

The vocational evaluator positions currently unfilled in the Central Region will be filled.

The vocational evaluator position in the Southwest region will not be filled.

A new position has been created. The position will supervise the vocational evaluators, the business development managers and the self employment enterprise staff. Duties of this position will include:▪ Updating the DARS Best Practices Manual▪ Developing improved labor market data for use by DARS▪ Developing requirements for report writing▪ Procuring new instruments identified in the literature review and

conducting training

30

31

Clearly communicate the purpose of the program evaluation to staff involved. Describe the role of the program evaluator. Provide a program evaluation plan as soon as possible to the stake holders. Include the stakeholders in the planning meetings if possible Present the findings to those involved and allow for input before final

evaluation report. Don’t take comments personally. This is the nature of the job.

32

www.decisionswithdata.com Blog

33

Null Hypothesis (Ho) and Alternative Hypothesis (Ha) Stated

Population Distribution Determined For Variables Probability plot

▪ Normal Distribution Non-normal Distribution or small sample size

Parametric or Non-Parametric Test Chosen Test of Difference Between Proportions▪ Z-test: parametric▪ Chi-Square Test: non-parametric

Test of Difference Between Means▪ T-test: parametric▪ Kolomogorov-Smirnov Test: non-parametric

See Appendices of Program Evaluation for statistical formulas

34