THE EVALUATION OF CRITERIA & PHASES OF EVALUATION

Jose Paul Joseph

THE EVALUATION OF CRITERIA To evaluate its training program , a

company must decide how it will determine the program effectiveness ; that is it must identify what outcomes or criteria it will measure.

D.L Kirkpatrick's four-level framework for categorizing training outcomesLevel Criteria Focus

4 results Business results achieved by trainees

3 behavior Improvement of behavior on the job

2 learning Acquisition of knowledge, skills, attitude , behavior

1 reactions Trainee satisfaction

CONTD.. Kirkpatrick's frame work suggests that

higher level outcomes should not be measured unless positive changes occurs in lower level outcomes.

Also framework implies that changes at a higher level (e.g.: results) are more beneficial than changes at a lower level (e.g.: learning ).

Framework has been criticized for number of reasons first research has not found that each level is caused by the level that precedes it in the framework.

CONTD.. Second the approach does not take into account

the purpose of evaluation. Third, use of the approach suggests that

outcomes can & should be collected in an orderly manner, realistically learning measures need to be collected at approximately the same time as reaction measures , near the end of program.

As a result of these criticisms, both training practitioners & academic research have argued that more comprehensive models are needed.

Accordingly training outcomes have been classified into five categories : cognitive outcomes, skill-based outcomes, affective outcomes, result & return on investment.

COGNITIVE OUTCOMES Cognitive outcomes are used to

determine the degree to which trainees are familiar with principles, facts, techniques, procedures or processes emphasized in the training program.

It measures the knowledge trainees learned in the program.

Cognitive outcomes are level 3 criteria in Kirkpatrick's framework.

Typically ,pencil-paper tests are used to assess cognitive outcomes.

SKILL BASED OUTCOMES Skill based outcomes are used to assess

the level of technical or motor skills & behaviors.

Skill based outcomes include acquisition or learning of skills and use of skills on the job (skill transfer).

Skill based outcomes relate to Kirkpatrick's level 2(learning) and level 3(behavior).

The extend to which trainees have learned skills can be evaluated by observing their performance in work samples, skill transfer is usually determined by observation.

AFFECTIVE OUTCOMES These are outcomes related to reaction,

motivation & attitude. One type of affective outcome is trainee’s

reactions towards the training program. Reaction outcomes measure the trainees

response to a training event such as training facilities, trainer performance etc. They are also known as trainer evaluation.

Reactions are useful for identifying what trainees thought was successful or what inhibited learning.

Reaction outcomes are level 1 criteria in Kirkpatrick's framework.

CONTD.. Reaction outcomes are typically

collected via questionnaire completed by trainees.

Other affective outcomes that might be collected in an evaluation include tolerance for diversity, motivation to learn, safety attitudes & customer service orientation.

Affective outcomes can be measured using surveys.

Affective outcomes relate to Kirkpatrick's level 2 ( learning) or level 3 ( behavior) depending on how they are evaluated.

RESULTS Results refer to overall benefits of

training for organization. Result outcomes include improved safety at work site, increased production, decreased in customer complaints etc.

Results outcomes are level 4 (results) criteria in Kirkpatrick's framework.

RETURN ON INVESTMENT (ROI) ROI refers to comparing the training’s monetary

benefits with the cost of the training. ROI is considered outside of Kirkpatrick's four level

& is often referred to as level 5 evaluation. Training costs can be direct & indirect Direct costs include salaries & benefits for all

employees involved in training , instructors, consultants, program material & supplies etc.

Indirect costs are not related directly to the design , development or delivery of the training program.

Benefits are the value that the company gains from training program.ie; reduced costs or increased revenues.

In many cases ,benefits can only be estimated.

PHASES OF EVALUATION PRE-TRAINING EVALUATION Identification of training needs Evaluate performance standards of the

trainee Evaluate training objective Input evaluation EVALUATION DURING TRAINING Observation Behavior analysis Course audits Session assessments

CONTD.. POST-TRAINING EVALUATION Reaction evaluation Learning evaluation

IN DETAILIdentification of training needs: if the line manager is actually interested &

owns the responsibility of developing his staff, he would monitor their performance & identify their strengths & weaknesses.

If the deficiencies demand immediate correction, then he may not even wait to respond to his training needs.

Sometimes training needs get identified when the organization has to respond to

Declining company image Change in technology/product/process Introduction of new system/policies/procedures

CONTD..Evaluate performance standards : Directly linked to training needs is the

standards of performance. This information helps in setting realistic

objectives. While needs of the trainee are person

specific ,performance standards are job specific.

Performance standards identified as KRAs finalized between the individual & his boss.

CONTD.. Evaluate training objectives: this is to assess

whether there is a goal congruence between the trainee's needs & training objective.

Evaluate trainee’s profile: measuring pre training knowledge, skills & attitude is desirable to compare them with post-training performance.

Input evaluation: input evaluation gives an opportunity to the trainer to put things right before the package becomes fully operational. Committee approach has been found to be fairly successful for input evaluation. Brainstorming session would also help in situations where feedback on the subject is already available.

CONTD.. Observation: trainer’s perceptiveness &

observation skills cannot be discounted. Behavior analysis: usually assessment

during training through behavior analysis is relevant for imbibing of certain skills & attitude for example leadership skills, team orientation & attitude towards learners.

Course –audit: audits can be conducted to give feedback for immediate course correction

At the end of each day Mid way through the course

CONTD.. Session assessment : Assessing each

session at the end or assessing each session at the end of the day is possible. Normally this is done by using semantic differential scale, three point or five point.

Reaction evaluation : at the end of the course ,course reactions are collected by the use of rating scale. Rating scale are forms on which at the end each training session/ programme, trainees are asked to place on a umber of five point scale.

CONTD.. Learning evaluation: the purpose of evaluation

at this stage is to obtain information on the amount of learning acquired during the training programme, irrespective of whether they go on to apply the learning on the job.

Knowledge learning: the purest form of evaluation of the knowledge learning is through tests conducted at each stages-multiple choice, objective tests or true or false questions. For open ended training (objectives not formulated as measurable),the best way to assess knowledge changes is to ask trainees whether their knowledge has improved .

CONTD.. Skill learning: in most forms of the

industrial training , skill objective are more important than knowledge objective. Since skills are acquired by actual practice , they can be best be evaluated by observing & analyzing actual performance of the trainees while they are practicing.

Attitude learning : a common way of evaluating attitude & opinions is to hand out a questionnaire at the start & at the end of the programme.