, DOCUMENT RESUME

ED 109 404

rrt

CE 004 238

AUTHOR 'Wallace,- Juanita D: , ,

TITLE A Comparison of Procedures for Evaluation ofVocational-Education Piograms. Research Series No.45.

..

3 ,

INSTITUTION' Tennessee State Board for Vocational t ucation,Nashville.; Tennessee-Univ., Knoxville. Occupational'Research and Development Coordinating Unit.

PUB DATE .1 Feb 75 ,

NOTE ..k' 35p.

k

EDRS PRICEDESCRIPTORS,

MF-$0.76 HC-$1.95 PLUS POSTAGE*Comparative Analysis; *Educational Assessment;Vducational Programs; *Rvalu&tion Methods; *Programvaluation; Research Project8; Secondary Education;

*Vocational Education . '

IDENTIFIERS' Rays Self qielckiist of Quality Vo, Tech Prograis; Site .

.

.Visits; Tennessee .. ,

. ..

ABSTRACTThe study compares two methods of evaluating

vocational programs in Teinessee: an on-site instrument. used by theTennessee State Division of Vocational Education .and aself-evalhation method using Ray's Self-Checklist of -Qualit'yVocational-Technical Programs. The study evaluates vocationalprograms in office occupations andgstributive$ agricultural, tradeand industrial, and home economics occupations education in 18,schools in 13 county and one,city,school system in Tennessee. On-siteevaluation-daia-were converted to numerical ratings according to the60 elements of the Ray'insiruient by'a panel of experts and'VerestatiSticallrcompared, 'element by elementc to numerical ratings onthe, 'Riy instrument ,completed by the local slhool profeSsionals. Sincethe Professionals and the panel;Of experts agreed on only 3 of the 60eieSits, the study concludes' among other'things).that informationobt4med from currently used j.nstruments for/on-site evaluations byvisli:ing teams was-not adeqfate to determipe ,relative quality ofvosWonal-te6hnical.programs: Several_reCommendations,for improvingthe? ay instrument as e.profitali,le,svalaation device are included., -

,

Te*pages of Appendixes'incliO4,,Rayis components,,for qualityvoc4tionalrtebhnical prograpt ,. the Ray teff-Che'gkliste, and aTennessee map c4 saool'S/stemS.lnvolved in'the study: (JR)

, .-. I, .-,,,'-': 7''' ,

, ,, , . , ,

, .

. . :. , ::Y- -- .,

. ******************4!*****4,********,**4* -***********,4************4*** , Documents acquiredibp ERIC inclUde-. many.fnformal unpublished '4

,* materials not availiPle.frOmeither:soirces. Mc Makes every effort **.to.obtaimthe,best.5opy, iv4la*e. -nevertheless, items -Of marginal *

' ;1. reproducibility are:-Often eilcountered.ind this affects the quality,: * 6f the,microgiche.a0 :Aard*opeYst.reprOdUiEtipns RRiCmakes available 40:

"'%.* via the ER.IC:DocumeivOeiroduction Service.(EDRSI. EDRS, is not . *

:::'?0,resporfiibte_for the !ilidlity of,:the,originiI.fticument. Reprodtictions-*--4-supplAJA-hy- EbRS.ard'tlhe'best that can, likiide from +he'original: *

.

i ili***************,**************************04******417***********.. . d ) ' I .:,1%,

,

!.p1,1I I's.

TENNESSEE

A PR 2 9 197g

RESEARCH COORDINATING UNIT UNIVERSITY OF TENNESSEE COLLEGE OF EDUCATION909 MOUNTCASTLE.STREET KNOXVILLE, TENNESSEE 37916

Research Series

o

a

CIS DEPARTMENT OF HEALTHEOUCATION ki WELFARENATIONAL INSTITUTE Of

EtA.C.ATIONBEEN .EPPO

It ,./F U f PON,

r OPINIONS. f.t F Ake; r QIPPE

i)TE Or

r

0

1'

-2

o

Research Series

No. 45

A COMPARISON-OF PROCEDURES ,

FOR EVALUATION OF VOCATIONALEDUCATION PROGRAM

10,sBy

Juanita D. Wallace

Tennessee Research Coordinating UnitUniversity. of Tennessee College of Education

,)and

Tennessee State Board'for Vocational EducationSam H. Ingram, Executive Officer

Nashville, Tennessee 37219

February , 1975 4

.6

4

a

This monograph constitutes the forty -fift1

pbblication in a series of studieri,.papers,

and reports on VoCational -Technical Educa-c

tion at'the Tennssee Retearch Coordinating

Unit. The study on which .the monograph is

based was conducted by JuanitaDePew.Wallace

while completing therecitgremants for a pc:-

toral degree in Vocational Technical Educa-

tion at The University ofrTennessee..

Pamela Moss, EditorTennessee Research Coordinating Unit

4

AI.

FOREWORD

The concern for evaluatidn of vocational education.

programs. wall evidenced in the 196Vocational Education

Act, Which suggested a five-year plan for evalUation Of

all programs. In 1971, the'Tennessee State Advisory'

ICounoil.reiterated the concern in a recommendation which

called for a'more comprehensive plan for evaluation of

vocational education programs ih,Tennessee.

Most methodologies used in evaluating vocational

education programs have, in the past, involved ,on -site

visits bj a team of consultants in addition to self-

evaluatidn. These procedures, used by the Tennessee

'State Division of Vocational-Technical Education, havea

proven-to be time-consuming and financially burdensome..

Thus, the need for an'in-de th study -on evaluation

procedures soon became obvioUs.

The purpose of this monograph is to present t

findings of a study which compare 1) the. evaluation

procedure utilizing op-site Pisits.. by a team of con-

sultants and 2) the Ray Self-Checklist of Quality

Vocational-Technical Programs developed by Dr. John

Ray at-.The University of Tennessee._

Z.

A glossary of terms and a bibliography of relevant,

sources have,been PrOvided for those interested in4

A

5

0

additional reading'. In an effort to gibe thd..reader

Amore information on the two evaluation 'syS'tes

compareslc the. following have been 4ncluded in the appen-

dices: 1) a list of components which CompriFe quality

vocational-technical programs, r) Ray's Self-Checklist,

and 3) a map\indicating:the geographidMcatiOn of

school systems involved in the study.

The ,intent of the monograph is tto offer pertinent

inform tion and facts on the' evaluation of vocational

education\programs. ft is hoped that the findings pre- .

A,sented will provide those interested iri quality programs

with a more comprehensive view of evaluation procedures., ..)

0

,

Garry R. Vice, DirecterTennesseejesearch Coordinating UnitUniversi* of Tennessee, KnoxvilleCollege of, Education

0

iv

It

1

Ar.

I

9C C

V

Al*

4

4

O

_

a

Se .TABLE OF CONTENTSp

Aw

Foreword4

Glossary of Terms.

Page--iii

vi

A

Introduction1

The Purpose and Objectives of'the Study '2

Scope and LiMitati,ons of the Study 3

Methods and Procedures4

Review of Related Liteiature

Ar,

Summary of Findings .......... . 8

ConOlusibhs10

,

Recommendations . .. . ...Recommendations RelitedSpecificaflyto the 'Ray Self-Checklistt

Recommendations Related to the Studyin General

12

Bibliogpaphy '

Appendices.. .. . . 17

, ..,

..A. Components Nor "Q ailt r7Vodplonal-Technical,Programs . - . . ... '.' . a

. i*... ( t

17'

B. RaySelf-Checkl,ist'of Quality Vocational-.technical Education Programs . 19

_ Tennessee Map of School' Systems Involvedin the Study

1 . fr 26I

V "6,

(

7

6.

A

GLOSSARY OF ERMS

The 'following terms were used in the study and are .

A

defined here for clarification.

,Quality vocational-technical program. .This is- a,

program which will provide .curret occupational) informa-

tion, enroll 'an interested, qualified student regardless

of his mental or physical potential), assess his needs and

,abilities, prepare him for successful entry-level employ-

inent and place,him in the occupational "cluster" for which

he., his parents', ,the. vocational teacher, guidance counse-

a

lors , and other' teachers have jointly agreed he should,

train.. "Quality" programs will always strive to satisfy

basic needs' and to develop job-getting, job - keeping'

jobadvancing skil is (Ray, 1973) .

. ,visiting, team. The- team is a panel of.

experienced vocational educators who visit and evaluate4

a. local schd ol vocational education program.r.

Panel of experts. This ,refers to persona from theA

field of vocational g-ducation state division personnel,

teacher educatdrS, and local directors/ of vocational

programs, who have served on 'at least twcf site visits'

and have at least three ,years of teachingsexperience.

The task of the panel of experts was, to convert the sum-_

,

many, of the state evaluation instruients to the Ray

Self-:Checklist evaluation..

0

viit

4mosimmoi0..'

*A"

,Profeasionals.. -Professionals Is a term used to

represent the' teacher, guidance counselor, and principal

in each of the participAing- local schools involved in

the study. The three above named persohs individually

comptkted the.Ray'Self-Checklist on tgt specified vocar

tional program in his school.

. Components. Components are those factors which are1 4

necessary in a quality vocational educational prograin ando

are lieted in Appendix A.'

Elements. Elements are ose'items which measure

more, specifically the Components and are listed in ,Appen-

dix B.

.1

P. ,

4

4,44

O

9

O

ti

4I. INTRODUCTION

Evaluation of.vocatiOnal-technical programs' has

become a' necessity for yocational, education leaders since

the passage of the Vocational Education Act Of 1963 and

the ensuing Amendments of 1968. The 1963.Act suggested

evaluation of 411 programs on a'five-year planrand-the

1968 4mendmdn'ts mandated it Concern about accountability'

in all of education also added to the emphasis on evalua:-

tion. it

In Tennessee during 1971, the State Advisory Council-.,

recommended the establishment .and implementation. of, a

comprehensive plan to evaluate programs on the basis, of

quality Itnd process. r Researdil had revealed that methdd=

ologies used to evaluate vocational education -programs

were inadequate. Most of those methodologies, whichr

required on-site visits by a team of consultants in addi-

. tion to a self-study or self-evaluation, were deficient

in providing input for planning,and-improving programs -\

(Ray, 1973). The on-site visits had proven to be time

W- consuming' and costly in terms of human and financial

resources.. The Tennessee State Division of Vocat2onat-

Technical Education utilized this type of evaluat#ng,

4procedure.

In order to effectively,plan and/or redirect voca-

tional programs, a thorn gh, systematic .and continuous

evaluatiOn procedure,wa needed (Starr, 1970). Thum, the

aa,

I

k 'i.4:" .

- . '.o..? ..

fA

f 'ttt

.V :.

,Zneed.forld more efficient way to evaltae vocational pro-,

grams' init r this study.,

nnessee was the basis '..

t;'.AskairdSult of the direction and priority'given to,----t 4i

.

ev4luatiOn Of vocational - technical education programs,

Dr.' John Ray (1973) initiated a ,stu through the Tennessee. 0 .

A

a a.

% "L,ReSeaKCh Cbsrdi ing Unit at The University of Tennessee .

iAi.:.. ,.-f .. 4. 4.

,D

to develop ai evaluation model for vocational programs at ,t, '10.,

the secondarVschOol,level in'Tennessee. Therefore-, the

two methods compared in 'the study were the method used by .

A

the Tennessee State Division of Vocational-Technical Educa-#

tion and the self- evaluation method using Ray's Self-Check-!

lisp of Quality6Vocational-Technlcal Programs.

II. THE PURPOSE AND OiJECTIVES OF THE STUDY

With the emphasis on accountability there was an. N.,.

,1:"

,

g4 increasing eoncern for program outputs and the importance'

,

.i.

.

.v 0

.of providing programs at the least possible cost to

sec5,ety.yThis con'cerni, along with the mandates of 19A3- .

, A

and 1968, illustrated the need for the development and .N

utifzation of the most efficient and effective SstOM o

evaltiation. . In the 1,971 recommendation, the Tennessee0

'State Advisory Couhcil'focueed attention on that need.

aRay concluded in his research that the procedure

used for evaluation must, not be complid4ted or consume

excessive amounts of human or financial..fesources. The

primary cause of the deficiency found in most p' pular

t

2

rr

Ct.r

, ea

,4 evaluation methods was the lack of identification and.

4 7 4 ,

verification of 'quality program indicators,:

In the4study: Ray developedfind,established a con-,' I

, . ,-sensus definition of quality vocational edUcation prOgramsr,t

,. ,, 6. - '1 ; AOPP

_ .> ... :

and identified a list of twentyessential components which,-

,.._

.yere expanded to form ,a sixty element checklist. ''Thise

became the Ray Self-Checklist. 'The model w*s designed -to

be easily tindeTstpod,by its users with attention given to .f4

,-- ,

the time required for use of the,ihstrument.', ..

e

Since the problem of the study was, to find a more s..,

. .

,. - e

comprehensive plan for' evaluating .vocational 'education '4 -

A .. .

programs in the State of Tennessee, it Was necessary to,, .

deteirMine whether a checklist system of, eya3Uation obtained'

..,

comparable results as a system using team visits for 'eva1- .

uating vocational programs. Iri comparing the two methods

of evaluation, two objective's were set forth;

1. .To cMtermine if the State on-ite team evaluation

instrument yielded the same evaluative res41tA, in terms

of quality indicators, as the Ray Self-CheckIiit yielded;

.2. To determine if there were ite included in the

Ray Self-Checklist that were not include, the on-sitete .rror-Yrrt

team instraffients.

1III. SCOPE AND IP1ITATIONS OF THE ,s7upy

0? 4i

a

,

The study 4encOmpaSSed those vocatioilal iirogramsY'in4 '

the thirteen county and one city sdhool 'Systeins which were

3

r

- ;

' . t.evaluated by. the Tennessee State Division of .Vocational-

0

'Ted.hri-ic0Eaucation system in 19711-73. The study included A.

Technical. Education. The e g teen schooiserepresente

prograrris4irftistributive Education, Office Occupatd,'_ons,

Agriculture, Education,.-Trade and Industrial Edudation, and

HoMe Edonomics Occupations Eduq,atCon. The sample consistedof 'those systems which requested a state vocational evalua-

,

-4- 'ition 7 during; fine 44691 t ye:ar".1:972-73. 1

.:Cdiheurriei.7and-lioniemaking Education and Health Education.

programs Were ricinciuded _as part of the ,study. The study'-.

was further limited to the evaluation system and procedures

included in.theState Division of Vocational-TechnibalEdUcation 'visiting .team evaluation process

' ,C .

Self-Checklist evaluation instrument.,

:IV., .METHODS AND PROCEDURES

and the Raye

4

Nr,

In the lattex. part of March, 1973, the eighteen .sum-. .

nary evaluation forms completed by the State evaluationteams

teamswere _obtained from thel ate Division of Vocational-.

4 .

thirteen county school systems and one city schodd System.

These evaluation repOrts were from schools which had been

evaluated by using official. evaluation instruments approved-

by the State Division of Vocational-Technical Education inp72 -73. The instrument used by' the sta, for the on-site

.visit was developed by a "spe'Oial task force .in the Program

,tPlanning Division of the State Division-of Vocational.-

,Technical Education.' Ten areas of vocational instruction

4

-

o

'''s-we.rQncluded for reaction of the} evaluators with respect, ,,to commendations,i suggestions, and recommendations which c''-i :',I-t",,' .Iwe re of a more 'subjective nature than the Ray Self-diecklist,:-.- ,.;.;

; , , f /-,,. , ..The Ray Self-Checklist rated sixty quality indicators on' a?. .

0 to 5 basis with 5 representing the highest possible value..In April, 1973, support and endorsement 'for involve-

ment in the study were solicited ';and obtained from local 7-/administrators in thirteen of the fourteen systems,,whicti

it , 1had particip.ated. in the state'oevaluatiOne., The thirteen\ / ,,.,systems provided a total of lifteen' schools ;for the etudik:

The one school system which did n,ot" choose tii,particjate, .

, (:,in. the study include.d three local/ schools. Duriqg the . .

first two weeks Of May, 1973,, the Ray- Self7CheCklipt, ina,,,tru7.';'..,ea ' / ,' / .

'meat was administered to 'each of the profeesjonals i(te c .,fiecti 's\ \,,. . ,'.-\\\... ........`- ,,,,,,,----

.--'...

principals, ancl guidance counselors) in-t't,he.fifteen %c,5,:cfSi'':.:' ..",''.:,-.:

, .

.

,

___,/__--.,---------', ...-/ 7 ,,',.

''.44.,' .111:;:,. of t::- '-';,... '1;!

N. s.;-$alf00Ii - InclUdid itT.theTstudy'. y ...-. : ./., , ...., ;i -1.,1,' i 141.

,.i./ --,- r.,,,,

,.

. V ": : ". . h IV I , :_ 1 ,,..:;..,: ;-'On*-:bO,Cation al -p,:ragram Pe r; schooI-7413 stele c,e'di,, 'f IW,';'.? .....

e

,

4

selecting the program,,g,ithitiy the: sichoi:51:,,the 'on yi*j41ri-On./1:2 , t 2

V

-'-' / :, , .1 .r. , '. -. Is :. ,- .,.., .,-.--.\\.,was to rive ,f4cir.,e uil :represeintat.3ioh from the triree ( ,. -- . , ..,-..,N \\,.;`-' - ''' / ,' ..,,,, . i tit,' ii -' t t 1 1.1.i. i ,

- 1,

- / ; . - /'-' `..,-.: ,- ..' ,(',; -, - i . ,, , -, : ; t i /.-/ i l'i ',' I' , ,..-.- . --t i 3. '.. 1"/T N., s,areas. Therig4.,h2a-1 .e4:411-te'in eqhdois i.inclu, eci ..six s4h:pizo; ,-. 7-. ;4//'- .- -.;/ . ,.'',,r1.:'1`.. fl '.' il .q,1'..,`:i 1 :7: .;N:.'

regicips.-of the /state. and the, "..rite--vocationa4.,seOlfcek:'

in each" o f 'the th;eje.:,;,tesi:.O40 t ate .1%- The .yoctitl'orral..-- . ,../` ' i .

,*4 t ,.*e. 4

/74.,42C,iC!"- it'it-4'..1 1 d 841:,

='-.4:.8%ificnafe:o ice!--ied*AtOti,orii!(;4-91radefaxl".":'/education :(4!), home 44Oitpiip43;;;e:ducatiOns

ilZ:# 1:44 43:04-aig uyivoi, ottidn.

ture "edUcation (5);;;1:`

PrO4 kaittit:%`1A r'/1 V41:4 S ,

4'1C'.1./ {3/ 7. *i! *14:, :.A

ti

,'.

A panel df.eXperts was ;elected with considerati-brigiven to equal.representation pf geographic location, ::_-

vocation,al'erVice .area, and..representation from "vocational../: ,

. :. teacher. educators, state department -,personnel 'arid,:.1Ocal ' - --'- ,. - 4 ,. .

, directori of vccationaLethication. '.In the East -Tennessee/,,. ,

. ,Of 1

7 i-region distrib'utive educations and 'home economics education ,/ ,--

,//were 'represented by a local directdr of -vocational educa-,. /

s

tion,,and 3. ,teaChe;r educator, respectively: The'Middle.. '. : , ', y/ / k i/ Tennesepe 'regiOnWas2.represented by a businets teacher

- ,-...

4-yt educato:rFianck- a local,,., of :trade and induste).-a1; edfl,-...-,- - ,: i,.:- ,':',,::.*-_, 5 -- .- 10 -.:.2): / ...,,cation:'- --4`;icical-,-,-teacher. of agricultUrea regitta/l. i/ #.,..:

. , , ......,superv.lor of bitistieSs edupation from 't5e:-.;s--tte/,i3.epa.r;.ment,. -.4. e . ...2. , ,- ....%. -..5. - / !,: r epreseniee a. ..tie:.4fes,t, Ten tie s s de: :re gi on ; -, ' 1

,,.....-.In -June' 197-3- it* meMbe4rs df the'

r

Pin,'-1 of- 'experts s-

. -we're-.-i.n',VItted University of TdrirreiSee at`-Knoxville

.

At the time of tillat! visit' the-experts wererequested to-'convert the sumniaiY5:144 -from the State Divi-sion evaluation summs-ry---rep:brti,forms -.to t... flay Self-Check-

,t -liat instrument. Only data ;.,4- *plied,by four- panelists whohadjthe gretteSt-degree- of

,agreemen were'i.neluded in, the:, . ..

. \.. .. ..,;\ -sy . ,, ..,\ tud

. ., .-. :: Y. :.:..\\Ai 'N'' ,. In .usin4 thep*O.,,,,,zinstra-.ratle-i reliability vitas estab-N\

... N'---- .'-.:'5:- '--:. '-'110=.t ...5... lished: T s was.ScComplashe'd by having each panelist con-.. :- "` s' -... - .

.. . e,,.., " .N .. . .- vert the summary data to the Ray' Serf-Che,dkliat on all

-5,.,- ,. 7 e . .) .Nschools inifolved,an-,-the study. and then 5by. cOMpUting Icendalq's,7,/ ,-''.-.,;. r -,z." s Coefficient of Concordance "W" (Siegel,. 1956) to determine-intra-rater reliability.

t,::' ,

.1

..,

-f,:--.,

-.--, t 6,..

.

, ,. ... i5:2 -,..,

5,

The on -site evaluation

ratings,on the ,Ray'Self-Che

of experts were compared, e

A c,

daa converted to numerical4011,4

cklist instrument' by the panel. . 4 .

lement tby'eleiit, to numerical.

1,111'

ratings on the Ray Self- checklist completed by the lekal

school professionals.

The intervals between means of 'the converted on-site10

evaluation data and means of the self-checklist da E%. were

determined and presented gratthically by sixty elements. A

chart: was used to plot the mean ratiAgs.

Student's "t" tests (Siegel, 1466) were used tC:deter--ks

mine the differences between panel and local school person-

nel ratings each eleme14:4.; The' "t "_ values were tested

'for sigilifiCanCe beyonthe .05 level.

V. REVIEW OF RELATED LITERATURE

The body' of literature related to evaluation of voca-.

tional programs contained many ,references identifying and,, .

establishing the need for evaluatiOn and accountability,_in

vocational-education programs. Program evaluation was

identified as a continuous process of gathering valid and

'reliable data for the purpose of assessing the extent to

which predetermined objectives and levels of.performance

have been attained. The major purpose identified for conZ

ducting an evaluation was to provide inforthation for makirig

sound, rational-choices and decisions'about vocational

education programs.,t

7

A

1

The literature stressed the need.for empirical evi-.

dence relating to need for quality vocational education-

programs in relation 'to expenditdres for these 'programs.

'Increased concern by the public and educatOrs not only fort

quality vocational progtams but also fOr'.

,involved.-

state go/%,-

andsacco

enditures

hasmade accountality a necessity. .Federal and

vernments have recognized the need for evaluation-At!

untabilitiby enacting legiglation which mandates

funds, and provides for evaluation of vocational education

programs.

The review of literature revealed a scarcity of formal

21esearch to substantiate .the deSign, validity, or use of.!

,eNialuation'methods or instruments. Many 'research studies.,

recommended a combination of v4 siting evaluation and--

. .

self-evaluation methods, but they provided no empirical

evidence, to support the need for-ilicluding both methods in:;

the Process.- No studies were found which compared the4

visiting team method or evaluation with-the self-evaluation

metho.'

-,t The literature indicated the complexity of the instru-

ments being used in evaluating vocational education program-S.

The time-contdming nature and costliness of the procedures''.- °

was frequently,stressed throughout the literature.

VI. SUMMARY OF FINDINGS

1. 'The professionals and the panel of experts indi-

cated basic, agreement on: only three .6f the 6.0.glIMent-k.

.8

*4,

$

2. /On 'three of the elements, the, panel of experts

wart unable to obtain enough information from on-site visit

reports to take any kind of judgment.

-3. Of a possible 900" observations, counselors wereI

unable to make observaticps on the 60 elements 155 times

(17.2%). .

A '1:44. Of a possible 900 observations, teachers were

unable to make observations on the 60 elements 28 times

5. Of a possible 900 observations, principals were

unable to make observations on the 60 elements 56 times

(6.2%),

6. Of a possible 900 observations, the panel of

experts ,was unable to make obsetvations on the 60 elements

, 347 times 138.5%).

. v.%

7: 'TheUpanel of experts and professionals agreed only

..

,

. -

on ,elements related to physical facilitieS. 4

.

8. The mean scores of all'profestionals on each ele-,

ment were consistently higher, and in most cases consider-,

Ably higher, than the mean scores given those eleMents by'

the panel of experts t,

' 9. Thg range and mean value of scores given on each. .

element yrie onsiderably mere on the pan 1 of expert

scores than it did with the professiolis ores.

10. All of the 'professionals were.ible to make obser-

k %1/4

vAtions on 19 (31.5%) of the 60 elements. ,

v.

.

9

11. Forty-five percent, of the time there were 20%. ar.

less response from the panelists.f,

4

,12. Seventy -five percent of the time there was 5'0% or

less response from the paneliSts: 4

13. Elements 22-27 had the lOwest response rate from

the profesionals, specifically from the counselors.4

VII. CONCLUSIONS

1. Information obtained from currently used instru- ..

ments for on -site evalUations by visiting teams was not. .

adequate to be able 'to determine relative quality of voca-

tional-technical programs,

2. Currently used state evaluation instruments were. . ,,

. 4not definitive ar specific enough to identify the same

'quality. aspects of vocationalyteahnical.progr as-di ..

A)

the Ray Self-Checklist. evaluation astrument.

3. prierfe-s-Aonal vocational-technical educators could

'not analyze teem Visit reports and adequately deter The- .

the quality q$, selected voca on-61,technical programs.'

State evaluation procedures n .to include checklists,

where more observations could be made.

4: Guidance counselors were less, able to identify

elements of quality Vocational-technical prbgrams than1

were teachers or.principdls.

5. SOrce of Vie elements on the Ray Self-Checklikt

overlapped in their interpretation.,

10

3

1.

t

P'

A

I '

6. A panel of experts, unfaMiliar with each Idi-)

,

vidual situation and trying to glean information from ari

on-site team report, was not able to identify outstanding

elements, of quality vocational programs as easily'as the

professional who `were closest to the situation.

VIII. RECOMMENDATIONS

The recommendations of the study were divided into

o-categories--those related specifically to the Rayfiti

Soltf-Checklist and ,those related to the study in general.

Recommendations Relted..Specifically to. the Ray Self-, Checklist

-.The followin recommendations were listed as a result

of the cvments and questions most consistently madevby

professionals and members of the panel of experts.- These-..

I,

:;"

-recomilhdations were not.based on specific data. Rather *-4/ ,-

..1

they were suggested as a result of the rcher's involve--. -,

ment and as iation with those, who rpspono&d to .thew instru7.40 ...-

ment.t!/ 4

.1: The "0" rating needed to be better *fined or ,.

.

omitted.

2. Element 6 needed to be refined so that it didmOt

confuse respor4ent.with',Eiemes 16, 17, and 18 (See

Appendix 8).

3. All elements needed to be stated cOnsistentlli.

either ;.n ,.the positive or the negative fOrm, specifically'

11.

/20

El Ment 48 which was, stated in the negative. The other1 i

ments appeared td be stated in thelpositive form.1%,

4. The instrument needed to be arranged so thai-a-T.

total score could be determined on each components in order,

that the component scores could be compared in-terms of a

quality profile.

'41,. A 0 .

. Recommendations Related to the Study in.General,-.

0

1.. Those-persons .responsible for conducting evalua-

tions of vocational-technical programs need to considerA

the results of this study for. the development of evalua-

tive instruments.

2. if team visit) pracesses "Are,to be d for eval-/I

uation, instruments should be revised to include more

. quality indicators, such as those included in the Raf 26

ol

;, Self-Checklist and shoUld be more objective in nature.

3. Ray's Self-Checklist should be subjec ed to fur.

ther validation by checking wits former students

employers of graduates of vocationalitechnical 1ograms .

Rays Sell--Checli4 should be further Validated

by havin4 visiting teams use the checklist in a school and'.

comparing the results of the team with those of the teacher,

.

guidance counselor, and-principaldn the same school.

5. An in-service training prograM for guidance coun-

selors should be provided to preParetheM to respond.more

adequdtely to the evaluation checklist.

J

1221

0

4

6. Since the-state evalUati8n instrument and Ray

SeLfChecklist do not identify ,,the same quality indicators

it ,is recommended that additional studies be' completed to\

further validate the Ray instrument. This further valida-

tion would in turn provide a,more firm basis for the c n-o

clusions and other recommendations for this study.o

13

6

tt.

t

BIBLIOGRAPHY

71 P

1. An, Overview of the Three Phase System for Sta LdeEvaluation of. Occupational Education Pro 'Am,S,.

Springfield: Illinois State Bciard of Voc tionalEducation and Rehabilitation, V',6Qational andkk

..,.Technical Division, ND. , ,

..',...a

, .. 4,,

. 2. Annual Evaluation Report. Tennessee State-,Ad i,oy. Council on Vocational Education %to ,the U.S.c,Com-

/ mission r ofiducation and the National Adv!

isory' Council on Vocational Educatici , ,1971,i !

. A3. Assessment Gu de lines for Programs Vpder the ,,Vocational

EdUcatio Act of 1963. Hartfd.d. !.ConneCticutState De artment of Education,4 Divission'of Voca-

vIt'

tional E ucation, January, 196 .

4. Bruhns, Arthur E. Evaluation Processes Used to Assessthe Effectiveness of Vocational,-Technical Programs.,Los Angeles': University of California, GraduateSchool of Education, December, ,1961.

5. Byxam, Harold M. Development and Demonstration Proceduresfor Evaluation of a Comprehensive Program of Voca-

" tional Education, in the Public School. East - Lansing:Michigan State University, College of Education, .

Bureau of EducatiOnal ResdOrch,Services,flOctober,1965.

Byrmr-'Harold M. Evaluation Systems for Local' Programsof Vocational-Technical Education: Final Report: 0

East Lansing: Michigan State Univerbity, College ofEducation, Bureau of Educational. ResOx!ch Services,October, 1968. ;

t ,

7. . Byram, Harold M. Five-State Try Out :and DemonstrationProgram to betermine the Generalizability of in

,

Evaluation System for Local Prolrams'of Vocationalan Technical Education: 'Final7Report. East

I Lansiqg: Michigan State UniverSity/College.of I

Education, February, 1971.' 0,i'l

. ...V

11r.'

8. Byram, Harold M. "Locally Directed lEy'al*tion of LdcalPrograms of Vocational Educaticint.' Ihstitute forImproving- Vocational Education:67: Final Report.Fayetteville: University of A'k7nsas, July, 1970.

.

...

,.

9. Byram,.Harold M. Locally Directed Evaluation of Local,Vocational Programs. East Lansing: Michigan StateUniversity; College of Educati0h, Bureau of EdUca-.tional .Research Services, Marc4, 1970.

' ,.

14

10. Characteristics of Quality Vocational-Technical EducationPrograms. Office of Education (DREW), Washington,D.C. : Division of Vodational-Technical Education, 4.

11.* eoste r, John K. The Role of the Sout hern' Association of___Colleves and Schools in Improving the Quality and a4

. -i

Quantity of ,Programs of Occupational Education.ip

Raleigh: North Carolina State University, Centerfor Occupational Education, 1968.

1,,.

A... .

12. Evaluation of Your Cooperative Distributive EducationProgram, Bulletin No. 1070. Baton Rouge: LouisianaState Deparpment of Education, Distributive EducationSdbtion,, 1966. .

13. Evaluative Criteria, Washington, D%C.: National Study of' Secondary School Evaluation, 1960.

14. Fourth Repeft--National Advisory Council on VocationalEaucation: Repptts on the Implementation of the

qational Education Amendments of 1968. Washington,Government Printing Office,` 1971.

15..,KIit, John A. "Evaluation'-the Key to Developing QualityVocational ProgramS,1:-Illinpis Vocational Progress.Springfield: Board ok Vocational Education and'Rehabilitation, Vocational and Technical EducationDiviSion, November, 1970..

16. 'Lessinger,'Leon. "Accountability," RCU 'Circulator.Knoxville: Tennessee Research Coordinating Unit for'Vocational Education, November, 1970.

17. ,MOss, Jeromer Jr. The Evaluation of Occupational Educa--tiOn Programs Technical Report: Minneapolis: Univer-sity of Minnesota, Research ,Coordinating Unit in..Occupational Education,.,September 1968A

18. National Seminar on Research in Evaluation of Occupa-tional Education. Raleigh: North Carolina StateUniversity,; Center for occupational Education, 1968.

19. Norton, Robert'E. Guide to Improving Vocational EduCa-tion Evaluation. Fayetteyille: University of Arkansas,College of Education, ,December, 1970.

Y

20. Norton, Robert E., Love, E.' Latac, ana Rolloff,'.j.ohn'A.Institute for'Impro,Ving Vocational Education Evalua-"tiOn:;-Finalport.- Fayetteville: University ofA rkaris

=Program Ana4S-is-Qtestionnaire.for'Vocational and Technicalducation'. yrovidence: Rhode Islands State Department ofEducation', Division of Vocational-TechniCal Education.

f.

b.

.

15

, .. .

24

Rajpal; Puran L. "Relationship.Between Expenditures andQuality Characteristics of Education in Public Schools,"Journal of Educational Research, October, 1969:

23. Ray, John R. Development of an Evaluation Model forTennessee Seconpiary Vocational-Technical EducationPrograms: Final Report. Knoxville: The University

. 'of Tennessee, Tennessee Research ,Coordinating Unit,January, 1973.

24. Reynolds, Harris W. Evaluative Criteria for Vocational-TechrliCal Programs. Harrisburg: -Pennsylvania StateDepa*Ment,of Public Instruction, Bureau. of Curriculum.

, Plannigand School'Evaluation, 1967.,

'25. Self-Evaluation. South Carolina State Departmpnt of.Edu-

f' cation, ,South Carolina Vocational EducatioOthedi,aCenter, 1970. sp

2f. Siegel, Sidney. Non parametric Statistics4qr the Behav-:ioral Sciences. (New York: McGraw-HilaBobk Company,

3 Inc. 1956.

Se. 4

21. Starr', ,Harold. A System for State Evaluation of Voca-tional Education: Final Repprt. Columbus: The Ohio.'State Universitpl,Center for Vocational and TechnicalEducation, May, 1970'.

28-. 'Vocational Educ ation Act of 196j-,' Public Law: 88-210. Walsh-ingt D.C.: Government, Printing Office,, 1963. ;

29. Vocational.*ducation Amendments of 1968/ Public Law 90-576.Washington, D'.C.': Government Printing Office, 1968.

4

.

30.* Vocational Education and Self - Evaluation. The VocationalEducation Instructor Appraises Product and Process. ,

J

Washington-, D.C.: American Vocational Association; r -.4:.

August, 1971. -, .. ,

.

a , el-31. Vocational Education Evaluation Summary_., Nashyille: 'State

Department- of Education, Division of VoCational-Technica1-Education, Program Planning and Evaluation Section, 1973.

Ward.., William Gary. Process Evaluation of VocationalEduz.cation: Review and Synthesis of Selected Researcp in

: Education Documents. Research Coordinating UnitlReportNo. 12. Stillwater:, Oklahoma-State'University, Decem-

. ber, 1970.

33. Wbodin, Ralph:J. -Guidelines.for the Deielopment of Instrti'..ments for Evaluation in Vocational Agriculture.'COlum=::'bus:; The Ohis!? State University, The Center for Vooldr!tional Education, 1966. ,

F 4; r6i

34. Wyllie, Eugene Donald. An Evaluation Plan for Business'Education Programs in High School's. SouVI-Western Mono-'graphs in Business and Economic Education', 4o, 109.Dallas:. South-Westerh.Publisbing Company, 'September, 1963.2516

op

APPENDIX A

COMPONENTS FOR "QUALITY" VOCATIONAL - TECHNICAL PROGRAMS*

1. Compliment, not conflict with the objectives of the total educa-tional program. ,

e. 41/4

410

t 2. Haveiberbd1, visible support of administration, guidance counselors,..

nd other eachers (as much, as academic program.04.

3 'Provide broad-based testing program and vocatibnai'guidangeid) .

seiirices so that each student can select a proper vocational.program. . 0 ,-,

.. .t-4-

. 4c. , . ,

.4. Arrange, 'parent-student- teacher conference'when'studeut is ertrolildand throuhput 111.73 prqgrdm, if feasible. ,1 .

. - ...

. . .:-...

,....

,..

5. Provide flexibility fors student, to move within program if Sneed .1) c

Ind/or ability indicat4 a move is advisable.`-. ..

6. Provide physical facilities, equipment and instructional' ..-

4fterials appropriate to occupational "cluster" for whiqh' student vpis being trained.

,'

. -.

. ,.

.

t.a v . . 0. ,

7: Dst only teachers who have completed state certification requir c,-.°'-- .. -.

ments for area in which they teach, who ba.lie recent related "X.../-,

.re eo, .

occupational experience, who are proficient in AkiLlpsthey . ;.

t -tea4, and who. engage in professional growth activities.

8'. Proy'ide cooperative or directed o9typational experience -ind/or 0'.

. .

aimulstsd occupational expe\lendi: A.

9. Actively use "representative advisory committees in occupationalneeds assessment, program planning, and evaluatida.

',10. Conduct periddic follow -up surveys of former students and keep

.. up=to-dair records for use in. program planning and'evaluatioa.

,..

_ ,

11. Make periodic surveys of employment opportunitie6 and use findingsto keep, curriculum releant.l'o

.

12. Have plan of continuing and systematic' internal evaluation for,'improvement and development ofprdgrat. ..

.,

. .

4

13. Endorse. and implement active lea4ership development program throughrelated vocational youth organizations:

, .' ..

,

. .

, . ap 4.

*Ray, 1973.

Pr

.0.14. Develop proper attitudes toward work, good employer-employee

relatiOnships, and efficient work habits and create the desireto continue personal and occupational development.

15. Stress each student 's total development within his mental andphysical abilities.

16. Provide for satisfactory completion of defined program for atleast 90 percent of studentsi./11(lo,not!transfer or withdrawfor health reasons.

17. Place students at thecompletion of their- programs in the. occupational "cluster" for which they are trained.

yl18. Make an organize&sdheorplacement service-available to graduates.

19. Keep public informed--have "participatory" support of community..

20. Use any available means to acquaint students'at all levelswith vocational-technical career opportunities.

A

18

ly

p1.

4

. ,-;.5.*4>

e: '. . ,1 :"

/Vs:. t.):Ifr; ,

, , . -4. I

I.

- . ?:-

r

10,

RAY SELF-CHECKLIST Oiv,' QtfAlpri VOCAV,IONAL-TTEciiNICAb EDUCAtOli PROGRAMS

a:, ,.'.. ', fr ... . 4 %

: 1 " ' i. ...4

1

School, , - e- Program Area ii-1

1,.

Teacher Principal Guidance CouittJ../Or,

.-:.Please rate frankly your vocational,t;technical program usf:nethe folloWirng

if- .:, ,. N

scale: .V.: f

t V,-, ; ,/ , S Exce:14e nt.;:yery well done :

4 a

4 'Sati-Ef-KforY; adequate . ,

, , .

3 Sornef_liiiprovements needed; no, crucial "weakness (es)-/

41a.' a,

Z.., Wajcit *improvements neededr.:- -NO:-Of it made ./`

r . % X . , 0 No -op rtunity tio.pbseryel not applicable' ,-1% . - - --; _ . 1:

" .

-:- .-- -- . ^ e prograpr,objeCtives and reqnite-,

.

-.-:;,.. ",;:i..",,,,,..--ments are known and supporteti by'_,. --1,,- - > . -..- .- - the gat ulty and other students.

:%--, - .-;.,....;:f,-.--- ,..-- ,-7,..-- -4-;.----- 1 .

:. -- ,:; ,,,<_-,-,-,.."-,-14,44.t-ipnal-technical students have,:::,/"..--l'afi';4111:iortunity to participate in

:--- .-- .1 , i , t..4-7:- ft k exgrat-c:urricular- activities.

--,..., ,i, . -,....

.":---;-.; , ' ..41.,,

.3...: kPro j AO* in your program are re- ,-..:c--,

'l.ate'd t..t0:--cther subject arta

.).. ... 1 - ' y. ej V:i11'

'...Teachetriand guidance counselors'4 i.,... .

-.- refer iri,A169sced' and '.. qualified\ , Siuderiiii--it,9",,,liocational-tqchnical

programs begIrdless of, gradeavecS6e. `.

tit c

5, Interacibn amorig. adni;irdstratorti,guidanc, counselorS,K vot.atfbnalteachers and otber",ieaCherscreates a-good learningenvironment ;

\

LI A 19'

5-_

-4--.

3 2 1 0

1... _

.

A

-

.

, 4'4 -`:- i:

vr, ..s...

.

1;.

.4( /'

,.-- .

s

- .

, c-..'4

....r.

. ...

,\

.

. -

..

i"

r.

1.- .

. .

.

..

..

. \\ .

...,k

.'.1

. ,

?!:

. C

0

6. Resources (time space / money) aresufficient for ah effective vocational-technical prOgraM to be conducted.

7. --Vocational counseling services helpstudents see thiii interests andaptitudes (as-well'as physical limita-

. tions),in light poccupationalchoices avaidiable:.

a. Vocational teachers; guidance coun-selors, and administrators systemati-cally examine the testing program forvocational students and make revisionswhere appropriate,.

9. Teachers and students review andunde7stand:tes1 results\as occupa-

nai plans are discussed.

lft.;40,11*-,Osits are Made to every irfter--

eateo\vocational-technical student;-prlar-io-enrollment.

DiScuSsion program objectives enablesthe prospective vocational studentand his parents to determine whetherthe objectives are in line With the' .

student's ability and_ occupational.,:objectives.

12."arent-student-teacher, parent=.:-

\:,teacher, and/or student-teacher con=-;5Ierences are summarized and filed..(i:',reated confidentially) for, future

use,

13. Student occupational objectives asdefined- in his program are periodi=cally evaluated-to determineadjustments are needed.

14. Changes in a student's progranjafterenrolled) can be made when consistentwith his performance and after a ;

codlierence with the teacher.

C,

,

20

.

4 3 2 1 0

, .

. .

)

-t

-/)

,.,..

..

.

.

.

.

1

N,

s

\ .

,....

.,

...-

-

-

,

f

1

r,4

9.9

15. Administrators and guidance counselorsassist the student in making prbgramchanges when he indicates such a

desire.

16. Tools, supplies,' machines, and equip-ment are-of sufficient number to con-

duct a quality program.'

17. Instructional' materials used in addi-

',tion to the textbookS are well

. selected, current, and'easily

^accessible.

18. Each student id provided adequateworking space and storage 'facilities

19. The vocational teacher interpretsthe program to the school and thecommunity and assists in creatinggood cbmmunity relations.

20. The teacher's skilI.',a0'content pre-paration for courses he teaches areof the type and quality supportedby the "representatiye" AdvisokyCommittee.

21. The teacher belongs to and partici-'pates in local civic or other.'similar organizations.

22. Occupational experiencl programs(cooperative, directed, or simulated)are provided for each student inrelation to hii occupational objec-tives.

,, ;23. Witten training plans' are developed '.

for each student's occupational'experience program in relation to r°

his occupational objectives.

24- Records are kept and.sulmarized onAthe teacher's regula visits totrainingAcStionsor on evaluationSof simulated occupaticlal experiences.

,

21

5 '4

. ,

3 2 1 -0

.

.

.

.

.

..

.

4

)

. ..

iI

1 I

.

,..A

_ .

.

r.

.

...4,

,

..

1 .

.

...

.

..

:40

25

s

An active Advisory Com mittee *re-,senting fields in which vocational -technical students are receivingtraining has been organized.

26. The AdvisorpoCommittee provides 'theteacher(s),:!!cdministratorg, and the

School Board 'information on currenttrends and devel5ppnts in the com-munity as they' rel4q, to vocational -technical programb%;

27. The "representative" Advisory Com-mittee asaistsUn providing currentoccupational information; helping

, place students and graduates,establishing standards and evaluatingthe program:

28. An annual follow-up survey of theprevious year's graduates is madefor the; purpose of deteimintng statusregarding employment. .

.-29. Daelcol.lected from follow-up studies.furnish evidence of how well voca-tional-technical educational objec-,,,tivei have been met and providea basis for. maintaining and improvingthe quality of vocational-technicalservices.

,30. Students are encouraged to reply tofollow-up requestsafter graduation.

31. A local occupational survey is con-- .

pleted annually to determilile employ-'

mentropport*nities available.

32. Curriculum content is current and inline with the latest equipment andpractices in the field of eniployment

22

5 4 3 2 1 0

. .

..,

, .

Ab

,..

...

A .

. .

.

...7.

......'.

.

7`- *-1

'

33. 'The skills, knowledge, and attitudesrequired f8T-employment in specificjobs are found in classwork; lab,experience program, and. youthactivities.

34. There is an on-going evaluation programof objectives, content, methods, out-comes, and student performance.

35. Students, parents, teachers, businesscommunity ,leaders, and.administratorsare involved in the annual program

/eval scion.

36. Findings from program'evaluation("feedback") bring about changes inthe curriculum-. .

37. Vocational yOuth organization, activi,

ties are an integral part of theinstructional program and the occupa-tional,experience program.

38. Membership and participatiOn in relatedvocational youth organizations areavailable to all students enrolledin the Oogram.

39. The activities of vocational youthorganizations are planned, imple-'mented, and evaluated by. students.

40. Each student is evaluated regularlyon his work attitudes, relationship ,with the'employei," work habits, andoccupational development.

41. Students exhibit genuipe,pride inquality workmanship.

-1 -

42. Instruction is provided :on humanrelations.

23 -111

1

- <

4

I -

.4.

43. Evaluation is made in terms of the.individual's progress-tcwArd his own'performance objectivesnot on thebasis of comparing his performancewith that of other students.

44. The vocational student's self-iimige andtotal school achievement are favorablyiffeCied by the vocational educationalexperiences.

45. Ethical practices and standards are.goals of the program.

46. A program of instruction is tailoredto the needs and abilities of theindividual students.

47. At least 90.percent of the studentswho do not transfer or withdraw forhealth reasons satisfactorily completetheir defined prograMS.

48. Fa5tors such as absenteeism, tardi-ness,. hehavioral problems, and costof supplies prevent students fromcompleting' their defined programs.

49. An up-to-date file on job'oppor-

? Lunities is maintaie'd and madeavailable to students in yourdepartment.

50. The "teacher actively assists graduate4in securing employment in the occupa-tional area forwhich they aretrained,

51. A high percentage of,graduates areplaced in the occupation for whichthey are trained.

52. An organized school placement serviceis_ available to all vocational

. graduates.

24.

44,

5 4 3, 2 1

*

0

. , .

i

.....

.

.

--'

.

,

..-

../.-

/. .

. .

s)..

}

t.

..:1

,

4

,p)

-r

53:4 Vqqational teachers'actively'seek the,'employment needs of the community.

.54. Students use sources other than theteacher apd/or school to find

entry-levhrjobs.

55. Your program has a favorable reputationin the community and among employers.

56. Th...:naverage" citizen knows aboutYoiir program and what it 'has to offer:

57. Employers support the prograi byrecommending it to other,studenis and

other employers,\donating equipment,furnishing consultants, sharinginformation and materials,'etc.

58. Vigorous efforts are made to insureSt-hat all students are informed ofvobatiolal-technical educational-Opportunitied andpeogram requirements.

59. Career occupation information hasbeen developed at the pre- vtational

grade level.

60. Resource persons from all occupationalareas are invited to assist inacquainting students with careeropportunities.

COMMENTS AND/OR QUESTIONS

A. 25

5.' 4 3 2 1 0

.A t

2.

f

y

.

,

.

..

..

A

1

.

f

,,,

6

0

a

,APPENDIX C

STATE OF TENNESSEE

9

.Agriculture

2.

Vocational Office

3.

Distributive

4.

Trade and Industrial

5.

Agriculture

6.,Tistributive

7.

Vocational Office

8.

Trade and - Industrial

9.

Vocational Office

10.

Distributi4e*

U.

Trade and Industrial*

12.

Agriculture

*Did tot participate in the study.

Figure 1.

Geographic location and

in the study.:-

to

4

vocational education

13.

14.

15.

16.

_Agriculture

.

Vocational Office

Distributive

Home Economics

17.

Agriculture

18.

Trade and Industrial

service area of school systems-involved