1

Relevance of Vocational Educational Training assessment for classes inindustry and Registered Training Organisations

H.J. Downing1, D. Humunicki2 and Z. Maric 21 Assessment Research Centre, Faculty of Education, University of Melbourne, 3010,Australia2 CSM Knowledge, 132-136 Albert Road, South Melbourne, 3205, Australia

AbstractThis paper describes a project that compared two forms of assessment of trainees. Theproject had two aims. First, to extend a trial of assessment materials that weredesigned for use in Vocational Educational Training (VET) in schools, to includecourses run by a Registered Training Organisation (RTO), namely CSM Knowledge.This was achieved by considering how many trainees were successful in satisfying aseries of sequences of criteria from two different units selected from trainingpackages. It was found that the results were not always consistent with perceivedlevels of difficulty as sometimes people who were not successful with the first, andapparently easiest criterion of a sequence, were successful with later, apparentlyharder criteria.Where the anomalies are caused by unclear descriptions of criteria, or by criteria thatare too difficult, these observations can be reported to the Assessment ResearchCentre as part of the nation-wide survey in which this current project is participating.The other aim of the project was to improve the feedback that was provided to theclient organisations whose staff were being trained. The usual form of feedback, inwhich trainees were rated as Competent or Not Yet Competent is very basic.More effective feedback has been provided by the self assessment forms that thetrainees completed before, and again after, training as they show the extent to whichthe trainees have improved their confidence and self perception.Another, more detailed report, based on the criteria-based assessment described in thispaper will provide the client with detailed information as to where the trainees couldbe effective employed. The report will also indicate what further training is required.Examining the number of trainees who are unsuccessful with particular criteria mayreveal deficiencies in the training and so suggest areas that should be improved.

2

1 IntroductionThis paper describes a project that compared two forms of assessment of

trainees. The project had two aims. First, to extend a trial of assessment materials thatwere designed for use in Vocational Educational Training (VET) in schools, toinclude courses run by a Registered Training Organisation (RTO), namely CSMKnowledge. Secondly, and more importantly, to see if the assessment materials wouldincrease the value of the reports that CSM Knowledge supplied to its clients.

2 Organisations involved

2.1 The organisationsThree types of organisation were involved in the project, a) the RTO, CSMKnowledge, b) two of CSM Knowledge's clients, "Client A" and "Client B", and c)the Assessment and Research Centre of the University of Melbourne (ARC). Theinteraction between these groups is shown in Figure 1.

Figure 1. Organisations in the project.

2.2 CSM Knowledge, the training organisationCSM Knowledge provides a range of educational services to government clients, aswell as to corporate clients, such as "Client A", and hospitals such as "Client B".Before recommending a training program to a client, CSM Knowledge systematicallyanalyses the client's training needs and produces a detailed report. This reportincludes, among other things, benchmarks, training needs of the client, and a range ofeducational solutions.Once the client and CSM Knowledge have agreed on the benchmarks and trainingneeds, CSM Knowledge works with the client to design a program that will suit the

3

client's requirements, while at the same time giving the trainees a tertiaryqualification.At the start of a training session, each trainee fills in a self-assessment sheet. At theend of the session, each trainee again fills in the self-assessment sheet, as well as anevaluation of the relevance to the employment of each of the training activities. Aftereach training session, CSM Knowledge provides the client with a copy of the trainees'self-assessment sheets as well as a report in which each trainee is rated as competentor not yet competent.For some time, CSM Knowledge had considered the benefits of providing clients withan additional report, which would show where the trainees, who had completed atraining program, were rated for a number of criteria.Such a report could help the client to decide where the trainees could make good useof their skills, and improve their job satisfaction. The report could also suggest whatfurther training would be most beneficial to each of the trainees.The present paper describes how CSM Knowledge investigated an assessment modelwhich had been developed by the University of Melbourne to meet the needs ofindustry and higher education. At the same time, CSM Knowledge participated in anation-wide trial of the assessment model.

2.3 University of Melbourne, developer of assessment model

2.3.1 Background to the assessment modelThe Australian National Training Authority (ANTA) commissioned a project topromote the recognition of student achievement in vocationally-based courses. Theaim of the project was to trial a model system that catered for the diverse approachesadopted in Australia.

2.3.2 Development of the assessment model and materialsThe project that ANTA commissioned started in 2001, when the Assessment ResearchCentre of the University of Melbourne developed an assessment model. TheAssessment Research Centre also developed criteria, assessment tasks, and scoringrules for units of competence in four industries, namely Information Technology,Manufacturing and Engineering, Hospitality, and Business Services.

2.3.3 Trial of assessment materialsThis year, 2003, a trial is being conducted to validate the assessment procedures andtheir use with the different combinations of assessment practices used throughoutAustralia (Griffin 2003).CSM Knowledge is participating in this trial while determining whether theassessment model is suitable for providing CSM Knowledge with detailed reports forits clients.

4

3 Stakeholder reporting needs and contextTable 1. Stakeholders, documents and formats used in the present project.Stakeholder Document Format

CSM Knowledge Administration Manual Hard copySoft copy down loaded

CSM Knowledge Resource Support Kit forTeachers

Hard copySoft copy down loaded

CSM Knowledge Student AssessmentRecord Booklet

Hard copySoft copy down loaded

Client Each of the above threedocuments

Hard copy

Client Explanations of project Oral communication inmeeting(s)

Client Reports on traineesa) Competent/Not yet

competentb) Trainees' self

assessment scoresCSM Knowledgemodel of assessment

c) Graded ratings fromAssessment ResearchCentre model ofassessment

a) Hard copy

b) Hard copy

c) Hard copyOral communication inmeeting(s)

CSM Knowledge Feedback from Client reCSM Knowledge's modelof assessment

Hard copy

Assessment ResearchCentre, University ofMelbourne

Evaluation of Unit(s) ofBusiness Services TrainingPackage by CSMKnowledge

Hard copyOral communication inmeeting(s)

4 Evaluation of competency unit(s)

4.1 Selection of units to be evaluatedCSM Knowledge selected the Business Services Training Package (BSB01) andInformation Technology Training Package (ICA99) as the training material to beevaluated as they were packages that were relevant to many of the clients. Fromwithin these packages, CSM Knowledge selected Units BSBCMN306A Producebusiness documents, and ICAITU126A Used advanced features of computerapplications, respectively, for the present project because of their relevance to clients,and because the units are part of the nation-wide trial that the Assessment ResearchCentre is conducting.The competencies and associated criteria are shown in Table 6 and Table 7.

5

4.2 Standard reference framework

4.2.1 Production of the standards reference frameworks used in this paperStandards reference frameworks are included in the Student Assessment RecordBooklets produced by Assessment Research Centre to accompany the correspondingTraining Packages. The frameworks for the two units that are studied in this paperhave been rewritten in point form for ease of reference. The resulting standardsreference frameworks are shown in Table 2 and Table 3.

4.2.2 BSBCMN306A Produce business documentsTable 2. Performance descriptions for each level of competence for unitBSBCMN306A.

Performance description Level

Design a range of documentsSelect the document most appropriate for presentation and the efficiententry of informationIntegrate documents into other work tasksUse input devices efficiently

4

Select the most appropriate software to optimise business documentsEnsure consistency of design and layoutEnsure efficient management of dataResearch and manage organisational requirementsUse accessible sources of information to solve problemsIdentify and order specialised equipment where necessary

3

Use a range of software to produce documentsAchieve consistency of designMeet the organisational requirements for speed, accuracy, style, layout, andmanagement of dataModify the physical work environment to meet OH&S requirementsStore and exit documentsObtain relevant information about applications

2

Use a limited range of software to produce documentsDescribe the organisational requirements for speed, accuracy, style, layoutand management of dataIdentify basic ergonomic requirementsIdentify how to store documents, and exit applicationsSeek solutions to problems from written references and/or help desks

1

6

4.2.3 ICAITU126A Use advanced features of computer applicationsTable 3. Performance descriptions for each level of competence for unitICAITU126A.

Performance description Level

Explore range of processes to enhance computer applicationsAnticipate and reduce difficulties with computer applicationsDevelop process to analyses problems with computer applicationsAutomate customisation and optimisation of PC environment

4

Suggest ways to improve transfer of data between applicationsSuggest procedures for solving routine problemsResearch updates and advanced technical solutionsAutomate routine activitiesEvaluate effectiveness of PC environment

3

Use advanced features to prepare, present and transfer dataAutomate and simplify routine activitiesAccess and use available support resourcesConfigure and customise PC environment

2

Describe and compare different data formatsExplain nature and functions of objects, macros, and templatesSpecify how outcomes can be met re tasks, customising and optimising thePC environmentAccess technical support services

1

4.3 Evaluation of the standards reference framework

4.3.1 Theoretical reference frameworksTwo of the best known theoretical frameworks for assessing students are those ofBloom (1974) and Bondy (1983).Bondy (1983) developed a criterion-based system for assessing the clinicalcompetence of trainee nurses with particular emphasis upon the amount ofsupervision the trainees required to perform their tasks.Bondy (1983) considered that the amount of supervision depended on the trainee'sprofessional standards especially the cognitive aspects, such as a knowledge base. Theamount of supervision also depended on the standard of the trainee's proceduresincluding accuracy, safety, effect, and affect.In this context, effect referred to whether the behaviour achieved its intended purpose,for example, if the purpose of a bath were to cleanse a patient's body, would the bodynow be clean? Affect referred to the manner in which the behaviour was performed,for example, did the nurse's tone of voice and demeanour reassure the patient.Although Bondy's criterion-referenced scale could be applied outside the medicalfield, the standard reference frameworks shown above resemble Bloom's frameworkrather than Bondy's. In particular, the frameworks in Section 4.2 resemble Bloom'smajor categories in the cognitive domain (Bloom 1974). There are six categories,

7

which in ascending order, are Knowledge, Comprehension, Application, Analysis,Synthesis, and Evaluation.A comparison of the four bands in the standard reference frameworks used in thepresent project, with the six categories in Bloom's categories in the cognitive domainshows the following approximate equivalents:

Table 4. Comparison of Bloom's taxonomy and the standard reference frameworksused in the present project.Domain (Bloom's taxonomy) Level in standard reference frameworks

Evaluation (No equivalent)Synthesis 4Analysis 3Application 2Comprehension/Knowledge 1

5 Methodology

5.1 Case studiesCase studies are the study of social practices in the field of activity in which they takeplace (Scarpens 1990), that is, the phenomenon under study is investigated within itsreal-life context (Yin 1989: 13). Experiments and surveys are excluded from thisdefinition since an experiment divorces the phenomenon from its context (Yin 1989:13). The primary means of collecting data in case studies tend to be interviews anddirect observation of activities.

5.2 Use of case studies in present projectIn the present project, case studies were conducted to observe the assessment of theachievements of two groups of trainees. One of the groups, from a commercial clientof CSM Knowledge, was being trained in production of business documents(BSBCMN306A), while the other group, from a major hospital which is also a clientof CSM Knowledge, was being trained in the use of advanced features of computerapplications (ICAITU126A).

5.3 ProceduresThe trainees in each group were assessed according to CSM Knowledge's method ofassessment. Each trainee was given a form to record their confidence in a number oftasks. After a day's training they were asked to fill in another section of the form toagain record their confidence in performing the same tasks.At the end of the training the presenter assessed the trainees' work according two setsof criteria. The first assessment was according to whether the students were deemedcompetent or not-yet-competent in the unit as a whole. The second assessment wasmade according to the range of criteria set out in Tables 5 and 6.

8

6 Results of assessments

In the first form of assessment, all trainees were deemed to be competent in thecompetent/not-yet-competent system. The results obtained in the second form ofassessment, that is according to their performance being assessed against specificcriteria, are shown in Appendix 1 and Appendix 2 and are summarised in Tables 5and 6 respectively.

Table 5. Results for trainees of "Client A" in the unit Producing business documentsusing the second form of assessment

Criterion Yesb Noc

1.1.1 Produce docsa with limited range of software 7 01.1.2 Use a range of software to produce documents 5 21.1.3 Select most appropriate from a range of software 7 01.2.1 Identify organisational requirements re docs 7 01.2.2 Select/prepare resources to design docs 6 11.2.3 Manage orgd requirements with other tasks 7 01.3.1 Identify ergonomic requirements of the user 7 01.3.2 Modify existing work space, furniture, equipment 7 01.3.3 Identify/order specialised equipment 7 02.1.1 Design document for entry/presentation of

information7 0

2.1.2 Develop range of documents and select appropriateone for entry/presentation

4 3

2.2.1 Describe org requirements re data management 7 02.2.2 Apply procedures for identification, opening,

generation or amendment of files7 0

2.2.3 Ensure efficient data management 7 02.3.1 Identify range of procedures for laying out docs 7 02.3.2 Demonstrate ways for achieving consistency in

design and layout7 0

2.3.3 Generalise procedures and use multiple functionsfor consistency of design and layout

6 1

2.4.1 Describe speed and accuracy requirements wheninput devices used

7 0

2.4.2 Demonstrate speed and accuracy when inputdevices used

6 1

2.4.3 Integrate efficient use of input devices into themanagement of other work

7 0

3.1.1 Describe org requirements for document production 7 03.1.2 Produce/cross check documents re org requirements 7 03.2.1 Identify task requirements for style and layout 7 03.2.2 Prepare/apply checklist for style and layout 1 63.2.3 Amend documents for style and layout 7 0a docs = documentsb Number of candidates (of total of 7) that met each criterionc Number of candidates (of total of 7) that did not meet each criteriond org = organisational

9

Table 5. (continued).3.3.1 Identify means for storing documents and exiting

applications7 0

3.3.2 Use procedures to manage storage of documentsand data integrity

0 7

3.3.3 Use procedures for managing storage of documentsand exiting from applications

5 2

3.4.1 Seek information on how to deal with problems 7 03.4.2 Use resources to prevent difficulties 7 03.4.3 Develop solutions to problems using limited or

readily available sources4 3

a docs = documentsb Number of candidates (of total of 7) that met each criterionc Number of candidates (of total of 7) that did not meet each criteriond org = organisational

10

Table 6. Results for trainees of "Client B" in the unit Use advanced features ofcomputer applications, using the second form of assessment

Criterion Yesa Nob

1.1.1 Explain demands and outcomes required 6 41.1.2 Use advanced features of applications 9 11.1.3 Explore options to enhance functionality of

applications5 5

1.2.1 Describe/compare different data formats 0 101.2.2 Select suitable data formats for applications,

And transfer data between applications10 0

1.2.3 Suggest how to improve transfer of data betweenapplications

7 3

1.3.1 Explain nature/function of objects, macros, andtemplates

9 1

1.3.2 Design and modify objects, macros, templates tocomplete routine activities

10 0

1.3.3 Create advanced objects, macros, templates toautomate routine activities

10 0

1.4.1 Display knowledge of application functionality andtools

0 10

1.4.2 Use shortcut keys to perform predefined operations 10 01.4.3 Create shortcut keys to automate user-defined

operations10 0

1.5.1 Use linking processes to automate routineoperations as required

10 0

1.5.2 Use linking processes to automate non-routineoperations as required

0 10

2.1.1 Describe routine problems 10 02.1.2 Refer to supplied resources to solve particular

problems10 0

2.1.3 Make suggestions on how to improve proceduresfor solving routine problems

0 10

2.2.1 Refer to on-line documents provided withapplication to overcome difficulties

9 1

2.2.2 Use search engines and discussion forums toanticipate and reduce problems

3 7

2.3.1 Refer to supplied manuals to solve problems 10 02.3.2 Explore external sources to identify updates and

advanced technical solutions8 2

2.3.3 Suggest to update manuals and training booklets 9 1a Number of candidates (of total of 10) that met each criterionb Number of candidates (of total of 10) that did not meet each criterion

11

Table 6. (continued).2.4.1 Describe problem, solution strategies and targeted

outcome9 1

2.4.2 Eliminate problems according to the results 10 02.4.3 Develop feedback processes to revise and modify

analysis according to results1 9

2.5.1 List and access technical support resources 10 02.5.2 Supply trouble shooting results and alert messages

to technical support10 0

2.5.3 Develop procedures that make trouble shootingmore efficient

0 10

3.1.1. Describe and apply a range of ways for enhancingPC performance

9 1

3.1.2 Generalise performance enhancement processesinto organisational and network systems context

0 10

3.2.1 Identify user and organisational requirements 5 53.2.2. Configure environment as required 6 43.2.3 Evaluate the effectiveness of the configuration 6 43.3.1 Identify ways in which PC environment could be

customised and optimised10 0

3.3.2 Customise PC environment to suit user needs 10 03.3.3 Propose techniques for generalising and automating

the customisation and optimisation across differentusers and PC environments

3 7

a Number of candidates (of total of 10) that met each criterionb Number of candidates (of total of 10) that did not meet each criterion

12

7 Discussion

7.1 Explanation of Tables 5 and 6The criteria in Table 5 and Table 6 are arranged in ascending numerical order. Eachunit is subdivided into elements which are further subdivided into sub-elements. Thelowest subdivision consists of the criteria. The distinguishing number for eachcriterion, as shown in Tables 5 and 6, reflect the three levels of subdivision. The digiton the left refers to the element of the unit. The digit in the middle refers to the sub-element. The digit on the right refers to the third level, the criterion itself and issupposed to represent the increasing level of difficulty within the sub-element. Theselevels of difficulty are related to the levels shown in the standard reference frameworkin Table 3 and Table 4.It is reasonable to assume that the number of trainees able to fulfil a criterion, shownin column "Yes", reduces as the digit on the right increases. Conversely, the numberof trainees unable to fulfil a criterion, shown in column "No", increases as the digit onthe right increases.The first step in the analysis of the results in Tables 5 and 6 is to look for anomalies inthe principle that the number of trainees in the "No" column increases as the digit onthe right increases.The next step in the analysis is to find an explanation for the apparent anomaly.

7.2 Analysis of Trainees from Client A in the unit Producing business documentsAs explained in section 7.1 the first step in analysing the results recorded in Table 5 isto look for apparent anomalies in the relationship between the sequence of digits onthe right of the criterion numbers, and the sequence of numbers of trainees in the "No"column.Anomalies with more than trainee were detected with Criteria 1.1.2 and 3.2.2.The scoring sheet for the unit Produce business documents (Appendix 1) shows thefollowing descriptions:Criterion 1.1.1 "Produce required documents using a limited range of software".Criterion 1.1.2 "Use a range of software to produce required documents".These descriptions raise the issue as to what is the difference between a "range ofsoftware" and a "limited range of software". The assessor must make a subjectivejudgement and this may account for the anomaly.The anomaly with Criterion 3.2.2 does not seem to be caused by any difficulty inmaking a judgement. There is no doubt what the criterion "Prepare and apply achecklist to ensure documents meet task requirements for style and layout." means.Furthermore, there is no reason for confusion between Criterion 3.2.2 and the criteriaimmediately above and below it.It therefore appears likely that the trainees had found this task difficult with only oneout of seven trainees being successful, because they had not been taught this in theirtraining session.

7.3 Analysis of Trainees from Client B in the unit Use advanced features ofcomputer applications

In Element 1 of this unit: Manipulate data, there are five subsections . In three ofthese, namely the first (Yes = 6, No = 4), second (Yes = 0, No = 10) and fourth (Yes =0, No = 10), the trainees experienced difficulties with criteria that were supposed to beeasy.

13

The criteria are described as follows:Criterion 1.1.1 "Explain the demands and outcomes required for a variety of complextasks and specify how package can be used to produce intended outcomes."Criterion 1.2.1 "Describe different data formats and compare their benefits anddisadvantages."Criterion 1.4.1 "Display extended knowledge of application functionality and tools."Each one of these is too difficult to qualify as a Level 1 in a standard referenceframework unless each was complemented by a scored assessment task. This was notthe case.There was also considerable evidence of too large a difference in difficulty betweenthe last criterion of a subsection of an element and the criterion before it. Examples ofthis are Criteria 1.5.2, 2.1.3, 2.5.3, and 3.1.2. For each of these criteria none of thetrainees were rated as successful (Yes = 0 and No = 10), whereas all of the traineessucceeded in fulfilling the criterion immediately preceding the difficult criteria (Yes =10 and No = 0). For Criterion 2.4.3 the fall in achievement is from Yes = 10 and No =0 to Yes = 1 and No = 9.

8 ConclusionThis paper has described a project that compared two forms of assessment of trainees.The project had two aims. First, to extend a trial of assessment materials that weredesigned for use in Vocational Educational Training (VET) in schools, to includecourses run by a Registered Training Organisation (RTO), namely CSM Knowledge.This was achieved by considering how many trainees were successful in satisfying aseries of sequences of criteria from two different units selected from trainingpackages. It was found that the results were not always consistent with perceivedlevels of difficult in sometimes people who were not successful with the first, andapparently easiest criterion of a sequence, were successful with later, apparentlyharder criteria.Where the anomalies are caused by unclear descriptions of criteria, or by criteria thatare too difficult, these observations can be reported to the Assessment ResearchCentre as part of the nation-wide survey in which this current project is participating.The other aim of the project was to improve the feedback that was provided to theclient organisations whose staff were being trained. The usual form of feedback, inwhich trainees were rated as Competent or Not Yet Competent is very basic.More effective feedback has been provided by the self assessment forms that thetrainees completed before, and again after, training. These are particularly helpful asthey show the extent to which the trainees have improved their confidence and selfperception.Another form of feedback, that is being developed for the clients, is a detailed reportbased on the standards reference assessment model described in this paper. Thisreport will provide the client with detailed information that will enable the client todecide where the trainees could be effective employed. The report will also enable theclient and CSM Knowledge to decide what further training is required to improve thetrainees within their current level, and also to improve their performances to a higherlevel.CSM Knowledge itself can also benefit by examining the number of trainees who areunsuccessful with particular criteria as this may reveal anomalies. These anomaliesmay in turn reveal deficiencies in the training being provided, and in doing so, suggestareas that should be improved.

14

9 ReferencesAnon 1 (2003) Student Record Booklet for (BSBOI) Business Services TrainingPackage.Anon 2 (2003) Student Record Booklet for (ICA99) Information Technology TrainingPackage.Bloom, B. S. (1974) 'An introduction to mastery learning theory'. in Block, J. H. (ed)Schools, Society and Mastery Learning, New York, Holt, Rinehart and Winston.Bondy, K. N. (1983) 'Criterion-referenced definitions for rating scales in clinicalevaluation', Journal of Nursing Education, Vol 22, No.9, pp 376-382.Griffin, P. (2003) 'Background' in Griffin, P., Gillis, S., and Jelinek, M.Administration Manual, A trial investigation of a Standards Referenced AssessmentModel for vet in Schools courses, Melbourne, Assessment Research Centre TheUniversity of Melbourne.Scapens, R. W. (1995) 'Researching management accounting practice: the role of casestudy methods', British Accounting Review, Vol 22, No.3, pp 259-281.Yin, R. K. (1989) Case study research: design and methods, (Revised ed.) NewburyPark California, Sage Publications.

15

Appendix 1 Performance Scoring Sheet for Unit Produce business documents(Anon 1 2003).

16

17

18

Appendix 2 Performance Scoring Sheet for Unit Use advanced features of computerapplications(Anon 2 2003).

19

20