Project ACCESSMeasuring Transition-Related Transition Indicators Among High School Students with Disabilities

Christopher Murray, Clayton Rees, Mimi McGrath-Kato, Jeri Dickinson

Oregon Office of Vocational Rehabilitation Services (OVRS) and The University of Oregon

Funded by U. S. Department of Education, Office of Special Education and Rehabilitative Services (OSERS)

Need For Project in Oregon

0

5,000

10,000

15,000

20,000

25,000

Adolescents withDisabilities

Adolescents withDisabilities in YTP

Number of Youth in OR

Other Needs

Majority of students who do participate in YTP receive services in Grades 11 and 12—our goal is to begin developing awareness of transition-related needs in 9th and 10th grades.

Special Educators and Vocational Rehabilitation Counselors both have strengths and unique skills. Our goal is to bring those skills together to provide students with a unique and meaningful experience.

To provide these experiences to a greater number of students—by targeting groups of students rather than individuals.

Overall Model Structure

State TeamInforms Local Team and

Sites of State Level ChangesAssist with Dissemination and Problem Solving

Local TeamProvides Direct Feedback to sites

Provides Feedback to State Team Regarding Implementation

SitesThree High Schools

Two Alternative SchoolsDevelop and Implement Model

Program

Participating School Districts

Springfield Public Schools--Semi-Urban

South Lane School District--Semi Suburban

McKenzie School District—Rural

We are attempting to develop models that can be applied in a variety of settings throughout the state.

Major Program Features To develop and implement a transition-related program for all students with disabilities in five

sites.

Place a VR Counselor in site 1 day per week (Year 1) to work directly with special education teachers developing and implementing model program. Time spent in schools should increase over project.

Teachers and VR counselors plan a curriculum/sequence of experiences for what information should be provided to students at grades 9, 10, 11, and 12 to assist them in making positive post-school transitions.

Working with families of students with disabilities, develop training model and provide parents with training.

This will be developed and implemented gradually to ensure program integrity…….

Year 1 Year 2 Year 3 Year 4 Year 5

Grade 9 Plan & Implement

Implement Implement Implement Implement

Grade 10 Plan & Implement

Implement Implement Implement

Grade 11 Plan & Implement

Implement Implement

Grade 12 Plan & implement

Implement

Step 1: Site-Based Services

Bring teachers and VR Counselors together at each site to develop curriculum that will be delivered to 9th grade students in Year 1.

We will provide “buy out” either through teacher extended day or substitute buy out for teacher planning.

We provide VR with funding to allow VRC’s to engage in the process of planning.

Next year VR counselors will spend one day per week at sites

One Example

Our Ideal Situation

VR Counselors are working with teachers. Providing co-instruction to students on the day when they are

there. We view this as an opportunity for VR counselors to provide their

insights regarding transition needs to students and teachers. We also believe that VR counselors will benefit and learn about

school processes that are designed to support the transition-needs of students.

VR counselors are getting to know students. This will provide counselors with better information for determining

which students could benefit from VR services. Teachers will continue to provide information to students as

much as possible when VR counselors are not in the school. This will help to ensure improvements in the delivery of transition-

related curricula and skills

Assessment process

Interested primarily on effects for students Want to have a standard assessment process across schools Want to have measures that are sensitive to change based on

the types of activities that are taking place in the schools. Want to have an assessment that measures the types of skills

that students will be learning in 9th and 10th grades Want assessments that are reliable and valid.

Meaning (a) internally consistent—reliable range from 0 - .99/1.0 with higher

being better (b) Valid in the sense that the measures are measuring the

constructs they purport to measure AND that the constructs are predictive of important post-school outcomes.

What we measure

Phase 1—baseline assessment of students, 2X per year The Career Locus of Control Scale Vocational Skills Self-Efficacy Problem Solving Inventory-Revised Outcome Expectations Scale Skills Rating System (Student Rating, Grades 7-12) Inventory of Teacher-Student Relationships School Bonding School Engagement (Behavioral Engagement) School Engagement (Cognitive Engagement) Employment Status Type of Employment Earnings

Assessing Functional Skills

Long-term Post-School Outcomes vs. outcomes that you get to see in school—intermediary.

Pathways of Influence

Annual GoalsChild Skills Intermediary Postsecondary

Indicators/ OutcomeFunctional Skill

How do you determine whether an intermediary indicator of skills is a predictor of later outcomes?

Phase 2—School Records, 1 X per year Race Gender Age (DOB) Grade level Disability Type Academic Achieve Grades Attendance Dropout status Diploma Type

Phase 2 Continued Teacher Data

Ratings on amount of curriculum covered, 3X per year Student Attendance in course devoted to ACCESS content

VR Data Ratings on amount of curriculum covered, 3X per year Number of referrals

School Demographics, 1 X per Year

What we measure

Phase 1—baseline assessment of students, 2X per year The Career Locus of Control Scale Vocational Skills Self-Efficacy Problem Solving Inventory-Revised Outcome Expectations Scale Skills Rating System (Student Rating, Grades 7-12) Inventory of Teacher-Student Relationships School Bonding School Engagement (Behavioral Engagement) School Engagement (Cognitive Engagement) Employment Status Type of Employment Earnings

The Career Locus of Control ScaleMillar, R. & Shelvin, M. (2007). The development and factor structure of a

career locus of control scale for use with school pupils. Journal of Career

Development, 33, 224-249.

Contains two factors, 1 = Strongly Disagree to 4 = Strongly Agree

Luck, 5 items Sample: “Luck matters more than careful planning in

determining whether I will be successful getting the job I want.

Internality, 5 items Sample “I believe through my own efforts I will be able

to be anything I choose”

Vocational Skills Self-Efficacy McWhirter, E. H., Raseed, S. & Crothers, M. (2000). The effects of highschool career education on social-cognitive variables. Journal of Counseling Psychology, 47, 330-341.

Contains one factor, 1 = No Confidence to 5 = Complete Confidence

Voc Skills Self-Efficacy, 36 items Samples: “How confident are you that you

could list four jobs in my general area of interest.

How confident are you that you could identify three of my strengths as a person.

Problem Solving Inventory-RevisedJaffe, E. B., & D’Zurilla, T. J. (2003). Adolescent problem solving, parentproblem solving, and externalizing behavior in adolescents. Behavior Therapy,34, 295-311

Contains five factors, 1 = Not At All True to 5 = Extremely True Positive problem orientation, 5 items

Sample: “When I have a problem, I try to see it as a challenge or opportunity to benefit from the problem in some positive way”

Negative problem orientation, 5 items “I feel nervous and unsure of myself when I have an important decision to make”

Rational problem Solving, 5 items “When I have a problem to solve, one of the first things I do is get as many facts

about the problem as possible” Impulsivity/Carelessness, 5 items

“When I am trying to solve a problem, I go with the first good idea that comes to mind”

Avoidance Style, 5 items “When a problem occurs in my life, I put off trying to solve it for as long as

possible”

Career Outcome Expectations ScaleMcWhirter, E. H., Raseed, S. & Crothers, M. (2000). The effects of high

school career education on social-cognitive variables. Journal of

Counseling Psychology, 47, 330-341.

Contains one factor, 1 = Not At All True to 5 = Extremely True

Outcome Expectations, 6 items Sample: “I will be successful in my chosen

career/occupation” Sample: My career planning will lead to a

satisfying career for me.

Social Skills Rating System (Student Rating, Grades 7-12)Gresham, F. M. & Elliot, S.N. (1990). Social Skills Rating System: Manual.

Circle pines, MN: American Guidance Service. Contains 4 Factors: 1 = Never to 3 = Very Often

Cooperation, 10 items Sample: “I listen to adults when they are talking to me”

Assertion, 10 items Sample: “I make friends easy”

Empathy, 10 Items Sample: “I try to understand how my friends feel when they

are angry, upset, or sad” Self-Control, 10 Items

Sample: “I ignore other students when the tease me or call me names”

Other Measures

Teacher-Student Relationships Communication, 8 items Trust, 5 items Alienation, 6 items

School Bonding, 7 items Behavioral Engagement in School, 5 items Cognitive Engagement in School, 10 items

Results: This assessment was administered to 245 students in Grades 9-12 from 5 high schools.

LD30%

OHI1%

ED9%

MMR4%AUT

4%CD1%

DD0%

VI1%

No Disability50%

Gender & Race

51.6 48

0102030405060708090

100

Male Female

Gender

64.3

28 6

116

0102030405060708090

100

Race

White

Black

Asian

Latino

NativeAmericanMultiracial

Other

Other Demographics

0

10

20

30

40

50

60

70

80

90

100

0

10

20

30

40

50

60

70

80

90

9th 10th 11th 12th

Internal Consistency Reliability of measures The Career Locus of Control

Scale Luck alpha = .83 Internality alpha = .76

Vocational Skills Self-Efficacy Alpha = .96

Solving Inventory-Revised PPO = .78, NPO = .76, RPS = .82,

Imp = .77, Avoid = .73 Outcome Expectations Scale

Alpha = .87 Skills Rating System (Student

Rating, Grades 7-12) COOP = .77, Ass = .75, Em = .80,

SC = .78

Inventory of Teacher-Student Relationships Comm = .88, T = .85, Al = .73

School Bonding Alpha = .82

School Engagement (Behavioral Engagement) Alpha = .82

School Engagement (Cognitive Engagement Alpha = .88

Criterion Related Concurrent Validity

Results of ANOVA’s Comparing Students Without Disabilities (ND) and Students With Disabilities (SWD) The Career Locus of Control Scale Significance

Luck ND < SWD, p < .001 Internality --------------

Vocational Skills Self-Efficacy ND > SWD p < .05

Problem Solving Inventory-Revised Positive Problem Orientation --------------- Negative Problem Orientation ND < SWD p < .001 Rational Problem Solving --------------- Impulsive/Careless ND < SWD p < .001 Avoidance ND < SWD p < .001

Outcome Expectations Scale ---------------

Social Skills Rating System (Student Rating, Grades 7-12) Cooperation --------------- Assertion --------------- Empathy --------------- Self-Control ND > SWD p < .05

Results of ANOVA’s Comparing Students Without Disabilities (ND) and Students With Disabilities (SWD)

IT-SR Significance Communication ND < SWD p < .05 Trust -------------- Alienation ND < SWD p < .05

School Bonding --------------

School Engage (Behavioral) --------------

School Engagement (Cognitive) ND > SWD p < .05

Tentative Uses

The underlying psychometric qualities of these instruments appear reliable and some evidence of validity.

Although many of the instruments are published and copyrighted, some are publicly available Vocational Skills Self-Efficacy Career Outcome Expectations Scale

Goals

Goals are tied directly to the results of assessment (or) current levels of performance and they are: Outcome Oriented Specify What, Specify by When, and Specify Measurable Criteria Goals do not specify what services will happen—that’s what the

rest of the IEP is for!

Important Issues In Determining Observable/ Measurable If the assessment contains norms you can look at standard

scores, percentile rank, on some metric (grade equivalency) when setting the goal.

If the assessment does not have norms, as is the case here, may to use another strategy such as finding the range of possible points and then setting a goal for growth.

Goals Continued

Example: By March 1, 2010 Jacob will improve his vocational self-efficacy skills from a score of 68/180 (38%) to 100/180 (55%) as measured by child-report on the Vocational Skills Self-Efficacy Scale.

Weak example: Jacob will improve his Vocational Skills Awareness.

Objectives

Ideally, objectives are tied directly to goals and are short-term monitoring mechanisms that allow you to determine whether the overall goal is being met. These are intermediate performance steps that

can be accomplished in two ways Break down the skills in the objective into discrete

components or skills, OR Describe the amount of progress within specific

segments of the yearly goal using the goal as a guide.

Objectives continued

Steps: 1) Split the year into three (or more) parts2) Take the overall difference in performance

specified in the goal and divide it into three parts. Present level = 68, Goal level = 100100 – 68 = 32, 32/3 = 11

1) Specify and list the performance expected at each objective time point

2) List the measurement that will be used to monitor the progress.

Objectives Continued

By March 1, 2010 Jacob will improve his vocational self-efficacy skills from a score of 68/180 (38%) to 100/180 (55%) as measured by child-report on the Vocational Skills Self-Efficacy Scale.

Ob1: By June 1, 2009 Jacob will improve his vocational self-efficacy skills by 11 points as measured by VSSE

Obj. 2: By November 1, 2009 Jacob will improve his vocational skills self-efficacy by 22 points as measured by VSSE

Obj. 3: By March 1, 2010 Jacob will increase his vocational skills self efficacy score by 32 points as measured by VSSE.

How can you measure progress toward goal? Give the measure multiple times (Practice Effects?) Divide large measures into 3 random parts—

meaning divide everything by three! 36 items / 3 = 12 randomly chosen items.

Now take the old range of scores, 36 to 180 and divide both by 3 to get new range of scores for each of your three tests.

36/3 = 12 (lowest possible), 180/3 = 60 (highest possible) If you wanted an 11 point increase at each objective time

point on the overall measure, divide 11/3 to get objective the new objective growth rate on each abbreviated test, = 3.6 or increase of 4 points at each time point.

Objectives Continued

By March 1, 2010 Jacob will improve his vocational self-efficacy skills from a score of 68/180 (38%) to 100/180 (55%) as measured by child-report on the Vocational Skills Self-Efficacy Scale.

Ob1: By June 1, 2009 Jacob will improve his vocational self-efficacy skills from 23 to 27 overall (4 points) on a 12 randomly selected items on the VSSE

Obj. 2: By November 1, 2009 Jacob will improve his vocational self-efficacy skills from 27 to 31 overall (4 points) on a 12 randomly selected items on the VSSE

Obj. 3: By March 1, 2010 Jacob will improve his vocational self-efficacy skills from 31 to 35 overall (4 points) on a 12 randomly selected items on the VSSE.