fasset skills development · web viewequipment and facilities - including all types of av...

Return on Investment (ROI) in Training

November 2010

Facilitated by

Faranani Facilitation Services (Pty) Ltd

The views expressed in this document are not necessarily those of Fasset’s.

TABLE OF CONTENTS

TABLE OF CONTENTS................................................................................................................. 1

ACRONYMS AND ABBREVIATIONS............................................................................................3

SECTION 1: UNDERSTANDING ROI...........................................................................................4

1.1 CONCEPTS FOR ROI................................................................................................................... 5

1.2 MODELS FOR ROI......................................................................................................................8

1.3 TYPES AND LEVELS FOR ROI....................................................................................................22

1.4 PROCESSES AND PRINCIPLES FOR ROI......................................................................................25

SECTION 2: APPLYING ROI METHODOLOGY..........................................................................28

2.1 ROI IMPLEMENTATION METHODOLOGY.......................................................................................28

2.2 DEVELOPING A PROBLEM TREE FOR ROI...................................................................................32

2.3 DEVELOPING A RESULTS CHAIN FOR ROI...................................................................................34

2.4 DEVELOPING INDICATORS FOR ROI...........................................................................................37

2.5 DEVELOPING A MEASURING FRAMEWORK FOR ROI.....................................................................39

SECTION 3: DATA COLLECTION AND ANALYSIS FOR ROI...................................................44

3.1 BASIC RESEARCH DESIGN AND METHODOLOGY FOR ROI.............................................................44

3.2 DATA COLLECTION OPTIONS FOR ROI........................................................................................45

3.3 DATA ANALYSIS OPTIONS FOR ROI............................................................................................46

3.4 EVALUATION TECHNIQUES FOR ROI...........................................................................................49

SECTION 4: ROI CALCULATIONS.............................................................................................53

4.1 VARIOUS APPROACHES TO CALCULATING ROI............................................................................53

4.2 TOOLS AND TEMPLATES FOR CALCULATING ROI.........................................................................61

SECTION 5: ROI IMPLEMENTATION CONCERNS AND MILESTONES...................................69

5.1 IMPLEMENTATION OF ROI – THE STEPS.....................................................................................69

5.2 CONCERNS AND CHALLENGES IN IMPLEMENTING ROI.................................................................75

5.3 IMPORTANT MILESTONES FOR ROI IMPLEMENTATION..................................................................80

ROI in Training (2010) 1

SECTION 6: ROI LESSONS FROM PRACTICE..........................................................................82

6.1 CASE STUDIES RELATED TO ROI...............................................................................................82

REFERENCES............................................................................................................................. 91

RECOMMENDED READING.......................................................................................................93

HANDY RESOURCES.................................................................................................................94

Useful links.................................................................................................................................. 94


Acronyms and Abbreviations

CEO Chief Executive Officer

CFO Chief Financial Officer

CIPP Context, Input, Process, Product

CIRO Context, Input, Reaction, Outcome

CRM Customer Relationship Manager

EEO Equal Employment Opportunities

ERP Enterprise Resource Planning

GST General Systems Theory

HR Human Resources

ICT Information and Communication Technology

MBA Master in Business Administration

OEM Organisational Elements Model

OSHA Occupational Safety and Health Administration

PDSA Plan, Do, Study, Act

RBM Results Based Management

ROI Return on Investment

SDF Skills Development Facilitator

SOE Standard Operating Environment

UCLA University of California Model of ROI


Section 1: Understanding ROI

“The ROI methodology is one that is replicable and scalable so that it may be collected and reported

in a cost efficient and timely manner and such that it is comparable and benchmarkable both within an organisation and external to the organisation.

2004 Global Learning Alliance

In the agricultural and industrial economies, companies needed "hard" workers, but the New

Economy is putting a premium on "smart" workers. The explosion of knowledge and technology

and the shortage of skilled workers have spawned an abundance of books and articles on the

topics of lifelong learning, knowledge capital, and intellectual capital. Accountability is a key issue

for Human Resources as well as other business units. Consequently, the idea of being able to

calculate the return on investment (ROI) of training is enticing. An absolute number in a neat

package, a trainer's dream! In some cases, it can be obtained. In other cases, while seductive, it

may not be worth the effort.

The more money a company spends on employee training, the greater the concern that these

highly skilled people will leave and take their knowledge somewhere else. This results in a loss of

knowledge and a poor return on the organization's investment in training. However, research has

shown that training actually reduces turnover and absenteeism. Employees will stay where they

can grow and develop.

Training in many organisations is not seen as the most popular activity. Why? HR and training

managers have to continuously justify the amounts of money that were spent on training. Training

is often seen as a waste of time, money and effort. The value added through training is

questioned. Rightly so. Assumptions are made regarding the value of training added in achieving

overall organisational effectiveness. How do we know the value of training if we do not measure

the value of training? ROI in training is about measurement. Measuring the return on the money

invested in training. ROI is about accountability and responsibility for the particular investment in

training.

The ultimate aim of any training program is to improve organisational performance that will add to

organisational effectiveness and profitability. In order to measure performance a person needs to

determine the monetary value of the performance in its current status. After the training

intervention has taken place, the learners' performance needs to be measured again, thus

determining if there was an improvement. Training is not a once-off event, but a continuous

process in achieving organisational effectiveness. Employees need to receive training on an

ongoing basis to be able to apply the learning acquired to their daily activities. The information


acquired to reinforce learning, will be converted to knowledge application once the specific task is

performed more efficiently and effectively. Only then can the ROI be calculated through a cost-

benefit analysis by determining the cost (investment in training) versus the benefit of the learning

that has taken place, i.e. the benefit of learning as a result of training.

Legislation has an impact on the way organisations view training, the purpose of training, and the

value of training delivered. South Africa has been investing vast amounts of money in training.

More so over the past eight years as a result of legislation in the form of the SAQA Act, Skills

Development- and Skills Development Levies Act. Training budgets are growing. Additional

training and development programmes are being implemented as part of organisations'

employment equity plans and workplace skills plans

The King Report on Corporate Governance states clearly that accountability must also be

reported in the area of human capital management. Training is an essential component of human

capital management. The good news is that ROI can be determined through a scientific method.

ROI is the measure of the monetary benefits obtained by an organisation over a specified period

in return for a given investment in a learning programme. In other words, it is the extent to which

the benefits (outputs) of training exceed the costs (inputs). The cost of training is the money

invested in the training programme. The benefits of the training programme would be the

measurable output, i.e. the ability to answer more telephone calls, or manufacture more nuts and

bolts.

Measuring ROI is not a one-person show. It is a powerful tool that enables training managers to

prove the value of training, gaining credibility for the value added, and a contribution to achieving

organisational effectiveness. It enables them to report to management in quantifiable terms, i.e.

rands and cents! Measuring ROI is about accountability and taking responsibility for the

measuring the impact as a result of the training. Measurement is about becoming a strategic

business partner that add value and provide integrated business solutions. Measuring ROI can be

done!

1.1 Concepts for ROI

If one looks at any facet of business the concept of 'return on investment' (ROI) is always a

relevant business topic. ROI can have many connotations depending upon the user’s perceptions

and motivations. In reality, ROI is really a measure of perceived value. Value can be different for

different stakeholders. Let's look at some examples:

An organization provides training to a group of participants. This person wants to know

the satisfaction levels of the participants.


A course designer creates an e-learning module. This person wants to know if the

module did its job in transferring new knowledge or skill to the learner.

A business unit manager sends two employees to training. This person wants to know

the impact the training has made on the job.

A senior executive measures performance by the business objectives that drive the

company. This person wants to know the degree to which training has helped drive key

business results.

The finance group manager views benefit relative to cost on every decision. This person

would want to know the benefit to cost ratio, payback period and ROI percentage from

training.

Value is inherent in each of the aforementioned examples. So the first question one should ask

when contemplating an ROI solution is 'How does my user of this information define value?'

Having said that, there is strong need to ensure that one has a balanced approach to learning

measurement. A balanced approach should be able to accomplish all stakeholders' perceptions

of return on investment.

ROI as process

ROI measurement is the process of collecting and analysing this performance data, and

translating this into a measurement of real financial benefit to the organisation. This benefit is

then compared to the cost of creating this benefit through training and measurement.

In many cases, ROI measurement can be linked to data collected and analysed for the purpose

of Training Needs Analysis (TNA). If detailed TNA studies are done prior to the training, the data

from these studies can be compared to the feedback and performance data acquired after the

training takes place. In addition, the TNA is likely to highlight the expected benefits and results

from the training. In this case, the change in performance may be more accurately determined.

ROI as perception

So, what actually is ROI on training? It can be considered to be a perception on the part of the

client of how valuable the training has been in achieving their perceived goals; and these

perceptions will vary depending on whom you talk to. For example:

The Board may see a big picture of how the training affects the company’s ability to

achieve its corporate goals

The finance department may be looking to see how training stacks up financially against

other ways to invest the company’s money, and whether the training, as carried out, was

financially more effective than alternative forms of development


The business unit manager may be solely concerned with the impact on performance and

productivity in achieving the goals of their department

The training and development manager may be concerned with the impact training

programmes are having on the credibility and status as the training function within the

company and its ability to secure investment in the future to drive further business

performance enhancements

Benefit and cost aspects of ROI

It is also important to understand that ROI can be realised in different ways. IBM has published

research on ROI, which points out the following factors to take into consideration. ROI can be

realised as:

“Cost avoidance”, which reflects the reduction in costs and overheads in various forms,

but which does not necessarily “improve” the value of the individual’s efforts to the

business (for example, training may reduce the time taken to perform a task, or attending

an online class may avoid travel expenses)

“Business benefits” are a reflection of what new and additional value (expressed as

business improvements) an individual brings to their work, or the performance of their

department or team, by virtue of their learning. (for example, learning may enable people

to perform tasks they were unable to perform previously, and provide valuable new

services)

By combining the 2, you can predict or calculate a total ROI on learning.

Why do we want to measure ROI?

ROI, and the evaluation of training, is, and always has been, an important topic to the computer

industry; and it has always been problematical convincing customers, and your own product sales

and marketing departments, that training is vitally important to the customer’s success with your

product. For the IT industry, training on new or updated software and systems is critical to

successful implementation.

Most people believe this, and intuitively accept it. Training is usually built into plans and proposals

alongside software purchases and other services. On the other hand, training is also often

regarded as one of the aspects of a project that can be cut back or even removed from a project,

without causing the project to fail completely; we are all familiar with the attitude that training is

often the last item to be included in the project plan, and the first to be cut back when money is

tight.

“Show me the value”

Many corporations, and corporate executives, are now more demanding in wanting to see, before

approving expenditure, a financial value justification for what training brings to the business, and


also wanting the actual benefit to be measured after implementation. Demonstrating ROI has

become a major point of emphasis for corporate HR development executives, who are facing

increasing pressure to justify their expenditure on people development, in terms of financial

benefit to the business.

“ROI” is a measurement technique common to many projects and investments, and can be

applied to training. It is necessary to be able to estimate, before the project starts, the potential

and expected ROI (using previous experience wherever possible), as well as be able to measure

and report on the actual return after the project has finished.

Looked at another way, ROI modelling also helps to support the argument that serious

businesses cannot afford NOT to train; the cost of not training (measured in various ways) can be

much higher and much more damaging, than the cost of training. For example, various studies in

various situations and circumstances have found some interesting statistics:

Untrained users take up to six times longer to perform the same tasks. Training enhances employee retention. A Louis Harris and Associate Poll says that

among employees who say their company offers poor or no training, 41% plan to leave within a year. Of those that say their company offers excellent training, only 12% say they plan to leave.

Studies show that in-house training costs 73% more than outsourced training.

A four-year study by the American Society of Training and Development shows that firms

who invest $1500 per employee in training compared to those that spend $125,

experience on average: 24% higher gross profit margins and 218% higher income per

employee!

Just a 2% increase in productivity has been shown to net a 100% return on investment in

outsourced, instructor-lead training

These examples are all well and good, but these are generic and non-specific indicators; how

well do these apply to any specific customer situation, or to a specific industry? If a customer is

going to make a decision to invest in training, they may want more specific supporting evidence

than this; hence the need to be able to offer a pre-training ROI analysis to give the customer

some comfort that this might in fact work in their situation.

1.2 Models for ROI

The measuring of return on investment originated in the USA, where the American Society for

Training and Development (ASTD) has been championing the process for decades. The ground

work was done by Donald Kirkpatrick who designed an evaluation system that addresses the

evaluation at four levels; reaction, learning, behaviour and results Reaction evaluation refers to

the typical evaluation forms we complete after a training course, the so-called smile-sheets in


which learners indicate the extent to which they enjoyed the course. The second level refers to

learning evaluation, in other words, the extent to which the learners have learned certain

concepts, principles, knowledge and skills. This type of evaluation is normally done by means of

tests because you can assess the knowledge level of a learner. The third level of evaluation is

called behaviour evaluation, and this type of evaluation seeks to determine whether people can

practically apply their knowledge and skills. Here we ask ourselves the questions: Is there a

change in behaviour? Can we measure the application of skills in the workplace? The last level

according to Kirkpatrick is results evaluation. This level of evaluation attempts to measure the

results in terms of the impact of the training intervention on the organisation. For example, are

there more sales as a result of a sales training programme? Is there an increase in productivity as

a result of a training programme? What is evident from all these questions is that there must be

some form of pre- and post course measurement in order to be able to answer these questions.

Jack Phillips, a training management consultant, is now considered to be the main expert in the

field of ROI in America. He visited South Africa in 2002 to share his ideas about ROI with local

training and human resource managers

More detail on the Kirkpatrick model….

Knowing there is a definitive need to measure the impacts of a large corporate cost like learning it

is fitting to have an industry acceptable model for doing so. This model is actually one that has

been in existence since the 1950's but continues to be accepted today using technology and

creativity to maximize its benefits for the modern corporation.

In 1959, Donald L. Kirkpatrick, author, PhD, consultant, past president of the ASTD and

KnowledgeAdvisors Advisory Board Member published a series of four articles called

"Techniques for Evaluating Training Programs." The articles described the four levels of

evaluation that he had formulated based on his work for his PhD dissertation at the University of

Wisconsin, Madison. Later, Kirkpatrick wrote a book (Donald L. Kirkpatrick, Evaluating Training

Programs: The Four Levels, 2nd Edition, Berrett-Koehler Publishers, Inc, San Francisco, 1998)

and it is now in its second edition. This book was a source for the information on the following

pages related to Levels One through Four.

Kirkpatrick's goal was to clarify what evaluation meant. The model clearly defined evaluation as

meaning "measuring changes in behavior that occur as a result of training programs." The model

itself is composed of four Levels of training evaluation. A fifth level, ROI has been added since

then. The fifth level was the brainchild of Dr. Jack J. Phillips, Ph.D., author, consultant and

KnowledgeAdvisors advisory board member and strategic partner. The illustration below and

subsequent commentary summarizes Kirkpatrick's Four Levels and Phillips' Fifth Level.


This grid illustrates the basic Kirkpatrick structure at a glance. The second grid, beneath this one, offers more detail.

Level Evaluation type

(what is measured)

Evaluation description and characteristics

Examples of evaluation tools and

methods

Relevance and practicability

1 Reaction

Reaction evaluation is how the delegates

felt about the training or learning

experience.

'Happy sheets', feedback forms.

Verbal reaction, post-training surveys or

questionnaires.

Quick and very easy to obtain.

Not expensive to gather or to analyse.

2 Learning

Learning evaluation is the measurement of the increase in

knowledge - before and after.

Typically assessments or tests before and after the training.

Interview or observation can also

be used.

Relatively simple to set up; clear-cut for

quantifiable skills.

Less easy for complex learning.

3 Behaviour Behaviour evaluation is the extent of applied

learning back on the job - implementation.

Observation and interview over time

are required to assess change, relevance of

change, and

Measurement of behaviour change typically requires

cooperation and skill of


sustainability of change.

line-managers.

4 Results

Results evaluation is the effect on the

business or environment by the

trainee.

Measures are already in place via normal

management systems and reporting - the

challenge is to relate to the trainee.

Individually not difficult; unlike whole organisation.

Process must attribute clear accountabilities.

Level One - Reaction

Per Kirkpatrick, "evaluating reaction is the same thing as measuring customer satisfaction. If

training is going to be effective, it is important that students react favorably to it."

The guidelines for Level One are as follows:

Determine what you want to find out

Design a form that will quantify the reactions

Encourage written comments and suggestions

Strive for 100% immediate response

Get honest responses

Develop acceptable standards

Measure reactions against standards, and take appropriate action

Communicate reactions as appropriate

The benefits to conducting Level One Evaluations are:

A proxy for customer satisfaction

Immediate and real-time feedback to an investment

A mechanism to measure and manage learning providers, instructors, courses, locations,

and learning methodologies

A way to control costs and strategically spend your budget dollars

If done properly, a way to gauge a perceived return on learning investment

Level Two - Learning


Level Two is a 'test' to determine if the learning transfer occurred. Per Kirkpatrick, "It is important

to measure learning because no change in behavior can be expected unless one or more of

these learning objectives have been accomplished. Measuring learning means determining one

or more of the following."

What knowledge was learned?

What skills were developed or improved?

What attitudes were changed?

The Guidelines for Level Two are as follows:

Use a control group, if practical Evaluate knowledge, skills, and or attitudes both before and after the program

Use a 'test' to measure knowledge and attitudes

Strive for 100% response

Use the results to take corrective actions

The benefits to conducting Level Two Evaluations are:

Learner must demonstrate the learning transfer Provides training managers with more conclusive evidence of training effectiveness

Level Three - Behaviour

Level Three evaluates the job impact of training. "What happens when trainees leave the

classroom and return to their jobs? How much transfer of knowledge, skill, and attitudes occurs?"

Kirkpatrick questions, "In other words, what change in job behavior occurred because people

attended a training program?"

The Guidelines for Level Three are as follows:

Use a control group, if practical

Allow time for behavior change to take place

Evaluate both before and after the program if practical

Survey or interview trainees, supervisors, subordinates and others who observe their

behavior

Strive for 100% response

Repeat the evaluation at appropriate times

The benefits to conducting Level Three evaluations are as follows:


An indication of the 'time to job impact'

An indication of the types of job impacts occurring (cost, quality, time, productivity)

Level Four - Results

Per Kirkpatrick, Level Four is "the most important step and perhaps the most difficult of all." Level

Four attempts to look at the business results that accrued because of the training.

The Guidelines for Level Four are as follows:

Use a control group if practical

Allow time for results to be achieved

Measure both before and after the program, if practical

Repeat the measurement at appropriate time

Consider costs versus benefits

Be satisfied with evidence if proof not possible

The advantages to a Level Four evaluation are as follows:

Determine bottom line impact of training

Tie business objectives and goals to training

Level Five is not a Kirkpatrick step. Kirkpatrick alluded to ROI when he created level Four linking

training results to business results. However, over time the need to measure the actual value

impact of training became so important to corporations that a fifth level was added by Dr. Phillips.

Dr. Phillips outlines his approach to Level Five in his book Return on Investment in Training and

Performance Improvement Programs, Butterworth Heinemann Publishers, Inc, Woburn, MA

1997. Dr. Phillips has written extensively on the subject, publishing or editing dozens of books on

the topic of ROI.

The Guidelines for Level Five are as follows:

Use a control group, if practical

Allow time for results to be achieved

Determine the direct costs of the training


Measure a productivity or performance before the training

Measure productivity or performance after the training

Measure the productivity or performance increase

Translate the increase into a dollar value benefit

Subtract the dollar value benefit from the cost of training

Calculate the ROI

ROI calculations are being done by a few world-class training organizations. They help these

organizations:

Quantify the performance improvements

Quantify the dollar value benefits

Compute investment returns

Make informed decisions based on quantified benefits, returns, and percent return

comparisons between learning programs

Dr. Phillips has created an ROI Methodology that he conducts certifications and workshops on

and has helped training organizations use the right tools to measure the ROI on organizational

learning. The methodology is a comprehensive approach to training measurement. It begins with

planning the project (referred to by Dr. Phillips as an Impact Study). It moves into the tools and

techniques to collect data, analyze the data and finally report the data. The end result is not only

a Level 5 ROI but also measurements on the Kirkpatrick 4 Levels as well. This yields a balanced

scorecard approach to the measurement exercise

While Kirkpatrick's model is not the only one of its type, for most industrial and commercial

applications it suffices; indeed most organizations would be absolutely thrilled if their training and

learning evaluation, and thereby their ongoing people-development, were planned and managed

according to Kirkpatrick's model.

For reference, should you be keen to look at more ideas, there are many to choose from...

Jack Phillips' Five Level ROI Model

Daniel Stufflebeam's CIPP Model (Context, Input, Process, Product)

Robert Stake's Responsive Evaluation Model

Robert Stake's Congruence-Contingency Model

Kaufman's Five Levels of Evaluation

CIRO (Context, Input, Reaction, Outcome)


PERT (Program Evaluation and Review Technique)

Alkins' UCLA Model

Michael Scriven's Goal-Free Evaluation Approach

Provus's Discrepancy Model

Eisner's Connoisseurship Evaluation Models

Illuminative Evaluation Model

Portraiture Model

and also the American Evaluation Association

A detailed model of understanding ROI according to this thinking and development

process is on the following page:


http://www.eval.org/

Diagram illustrating the ROI model

Source: Measuring the Return on Investment in Training and Development Certification Materials, Jack J. Phillips, Ph.D 2002


Another model that talks to ROI is based on the cost and complexity of the training that is

provided. The illustration below showcases a 3x5 learning measurement model to capture a

balanced scorecard of learning metrics that range from the low cost / simple solution to the higher

cost/ complex solution. Each may be applicable for different needs. Each is explained briefly

below.

Learner-Based

A measurement model that captures data from training participants at two distinct points during

the learning process. The first point is directly after the learning intervention (Post Event) where

the main measurement focus is on Kirkpatrick's Level I - and Level 2 to gauge satisfaction and

learning effectiveness. Because there is a high response rate to these data instruments it is also

critical to capture indicators for advanced levels of learning such as Level 3 - Job Impact, Level 4-

Business Results and Level 5 ROI. These indicators are in effect forecasting or predicting the

future impact the training will have on the participant and the organization.

A second data collection point is a follow up survey conducted a period of time after the

participant has been back on the job. This survey is meant to true up the forecast and predictive

indicators of Levels 3, 4 and 5 by gathering more realistic estimates now that the participant is

back on the job.

The approach is low cost if one leverages standard data collection instruments across their

training and utilizes technology and automation to capture process and report the collected data.

Thus it can be used for all of your training; each time a participant takes a class to yield

continuous measurements.


Manager-Based

This method has the same data collection points as the learner-based solution but adds a

manager-based dimension. The manager of the participant attending training is another important

data point. They can be sent an evaluation instrument timed when the participant receives a

follow-up. The manager survey focuses on Levels 3, 4 and 5 of the Kirkpatrick and Phillips

models therefore getting estimates surrounding job impact, business results and ROI from the

manager's perspective. The manager survey also asks 'support' type questions to understand the

on-the-job environment where the participant applied the training.

Due to the increased effort it takes to conduct and analyze manager surveys the cost and time to

measure at this level is higher than the Learner-Based approach. But, with automation and

technology to facilitate the dissemination, collection, processing, and reporting of the data, the

cost and time can be minimal. The result is that it could be used on a continuous basis for every

training event a participant attends. More realistically, it will be used on a periodic basis for more

strategic programs where manager data is more relevant.

Analyst-Based

This approach uses significantly more comprehensive post event, follow up and manager surveys

it also uses other analytical tactics that go beyond surveying. For example to analytically measure

Level 2 - learning effectiveness a detailed test is designed and administered to participants. Due

to the time commitment of conducting a significantly detailed data collection and analytical

exercise the Analyst-Based approach is only used for about 5% of all training programs in the

organization. Typically these programs are the more strategic or visible and have the budget to

afford a more costly and time -consuming measurement exercise.

Other ROI Models to be aware of:

Know that there are other models for describing ROI, such as:


The CIRO (context, input, reaction, and outcome) Approach

Stufflebeams's CIPP (context, input, process, and product) Model

Alkins' UCLA Model

Please note: it is not the intention of this document to provide an in-depth analysis of the various

models and their comparative merits. Suffice it to say that there are alternative ways to calculate

or describe ROI, and it is useful to know that they exist, and some of their characteristics. We

have selected the Phillips / Kirkpatrick model to work on, as it is well understood, and does lead

to specific financial numbers.


Stufflebeam – CIPP

Stufflebeam considers evaluation as the process of delineating, obtaining and providing useful

information for judging decision alternatives. The CIPP model of evaluation was developed by

Daniel Stufflebeam and colleagues in the 1960s, out of their experience of evaluating education

projects for the Ohio Public Schools District. Stufflebeam, formerly at Ohio State University, is

now Director of the Evaluation Centre, Western Michigan University, Kalamazoo, Michigan, USA.

CIPP is an acronym for Context, Input, Process and Product. This evaluation model requires the

evaluation of context, input, process and product in judging a programme’s value.

CIPP is a decision-focused approach to evaluation and emphasises the systematic provision of

information for programme management and operation. In this approach, information is seen as

most valuable when it helps programme managers to make better decisions, so evaluation

activities should be planned to coordinate with the decision needs of programme staff. Data

collection and reporting are then undertaken in order to promote more effective programme

management. Since programmes change as they are implemented, decision-makers

requirements will change so the evaluation activities have to adapt to meet these changing needs

as well as ensuring continuity of focus where appropriate in order to trace development and

performance over time.

The CIPP framework was developed as a means of linking evaluation with programme decision-

making. It aims to provide an analytic and rational basis for programme decision-making, based

on a cycle of planning, structuring, implementing and reviewing and revising decisions, each

examined through a different aspect of evaluation –context, input, process and product

evaluation. Stufflebeam viewed evaluation in terms of the types of decisions it served and

categorised it according to its functional role within a system of planned social change. The CIPP

model is an attempt to make evaluation directly relevant to the needs of decision-makers during

the different phases and activities of a programme.

In the CIPP approach, in order for an evaluation to be useful, it must address those questions

which key decision-makers are asking, and must address the questions in ways and language

that decision-makers will easily understand. The approach aims to involve the decision-makers in

the evaluation planning process as a way of increasing the likelihood of the evaluation findings

having relevance and being used. Stufflebeam thought that evaluation should be a process of

delineating, obtaining and providing useful information to decision-makers, with the overall goal of

programme or project improvement.


There are many different definitions of evaluation, but one which reflects the CIPP approach is

the following:

‘Programme evaluation is the systematic collection of information about the activities,

characteristics, and outcome of programmes for use by specific people to reduce

uncertainties, improve effectiveness, and make decisions with regard to what those

programmes are doing and affecting’

Stufflebeam sees evaluation’s purpose as

establishing and providing useful information for judging decision alternatives;

assisting an audience to judge and improve the worth of some educational programme or

object;

assisting the improvement of policies and programmes.

The four aspects of CIPP evaluation (context, input, process and outputs) assist a decision-maker

to answer four basic questions:

1. What should we do?

This involves collecting and analysing needs assessment data to determine goals, priorities

and objectives. For example, a context evaluation of a literacy program might involve an

analysis of the existing objectives of the literacy programme, literacy achievement test

scores, staff concerns (general and particular), literacy policies and plans and community

concerns, perceptions or attitudes and needs.

2. How should we do it?

This involves the steps and resources needed to meet the new goals and objectives and

might include identifying successful external programs and materials as well as gathering

information

3. Are we doing it as planned?

This provides decision-makers with information about how well the programme is being

implemented. By continuously monitoring the program, decision-makers learn such things as

how well it is following the plans and guidelines, conflicts arising, staff support and morale,

strengths and weaknesses of materials, delivery and budgeting problems.


4. Did the programme work?

By measuring the actual outcomes and comparing them to the anticipated outcomes, decision-

makers are better able to decide if the program should be continued, modified, or dropped

altogether. This is the essence of product evaluation.

The four aspects of evaluation in the CIPP model support different types of decisions and

questions – this is illustrated below in the table:

The CIPP model of evaluation

Aspect of evaluation Type of decision Kind of question answered

Context evaluation Planning decisions What should we do?

Input evaluation Structuring decisions How should we do it?

Process evaluation Implementing decisionsAre we doing it as planned?

And if not, why not?

Product evaluation Recycling decisions Did it work?


Kaufman promote an assessment strategy called the Organizational Elements Model (OEM)

which involves four levels of analysis:

1. audit or cost-cost - based solely on inputs, looks at reductions of cost between old and

new resources

2. products - the building blocks of a product or service within an organization that

contribute to the overall product or service (e.g., automobile fenders)

3. outputs - products or services that are delivered to external clients

4. outcomes - the value of the outputs (the aggregated products or services) delivered to

external clients and their clients and ultimately to society

Since the introduction of Kaufman's four-level OEM model, many researchers have used it as a

viable framework for evaluation. Others, though, have found it restrictive and have attempted to

modify and / or add to it. Kaufman, later added levels of impact that go beyond the traditional

four-level, training-focused approach which they felt did not adequately address substantive

issues an organization faces. Such modification to the model resulted in the addition of a fifth


level, which assesses how the performance improvement program contributes to the good of

society in general as well as satisfying the client.


Level Evaluation FocusSuggested

Levels*

5Societal

outcomes

Societal and client responsiveness, consequences

and payoffs.Mega

4Organizational

outputOrganizational contributions and payoffs. Macro

3 ApplicationIndividual and small group (products) utilization

within the organization.Micro

2 Acquisition Individual and small group mastery and competency. Micro

1b ReactionMethods', means', and processes' acceptability and

efficiency.Process

1a EnablingAvailability and quality of human, financial, and

physical resources input.Input

*Based on Kaufman's Organizational Elements Model (1992, 1995)

1.3 Types and levels for ROI

Delivery and accountability - Two terms that one hears mentioned with great frequency of late

and, perhaps for good reason. For too long now many of our organisations and institutions have

failed dismally to deliver results in line with expectations and to hold themselves accountable for

promises made or lack of performance. As the spotlight on performance intensifies universally,

the contribution made by the human resource development or training function will not be

excluded from this scrutiny.

In its most elementary form, the training responsibility in an organisation is not a complex

process. All the trainer needs to do is adhere to the following framework:

1. Identify the current and future key business imperatives impacting the organisation

2. Assess employee capability to meet these requirements or needs in terms of Knowledge,

Skill and Attitude


3. Design a training process to equip the employees with the necessary competencies to

perform at the desired levels

4. Implement the process using the most appropriate resources and training and performance

management technologies

5. Evaluate the outcome in terms of business results, reaction of the delegates involved, and the

cost of the training technology utilised.

Considerable organisational resource can be wasted if, (1) the training is not directly linked to the

key business needs of the organisation, and (2) there are no meaningful measures in place to

evaluate the efficacy, business impact or financial contribution of the training process, including a

Cost:Benefit or ROI analysis if appropriate. Every training intervention needs to be evaluated, but

not every programme requires the evaluation intensity of a ROI analysis. Only the high-cost,

organisationally pervasive training programmes would warrant this depth of scrutiny. However,

adhering to a set of basic ROI principles when designing any training process will always help in

ensuring positive results.

Training, like any key organisational activity, must be performed with professional discipline and

requires executive management stewardship if the full organisational contribution is to be

realised.

A model that specifically focuses on the LEVELS of ROI evaluation is the CIRO model.

This is explained below:

CIRO (context, input, reaction, and outcome)

The CIRO four-level approach was developed by Warr, Bird and Rackham.

The four components of evaluation

Adopting the CIRO approach to evaluation gives employers a model to follow when conducting

training and development assessments. Employers should conduct their evaluation in the

following areas:

C - Context or environment within which the training took place

I - Inputs to the training event

R - Reactions to the training event

O - Outcomes

A key benefit of using the CIRO approach is that it ensures that all aspects of the training cycle

are covered.

Context Evaluation here goes back to the reasons for the training or development event or strategy.


Employers should look at the methods used to decide on the original training or development

specification. Employers need to look at how the information was analysed and how the needs

were identified.

Inputs Evaluation here looks at the planning and design processes, which led to the selection of trainers,

programmes, employees and materials. Determining the appropriateness and accuracy of the

inputs is crucial to the success of the training or development initiative. If, for example, the wrong

types of learners were chosen to attend a customer care National Vocational Qualification

programme this would be a waste of time and money for the organisation.

Reactions Evaluation methods here should be appropriate to the nature of the training undertaken.

Employers may want to measure the reaction from learners to the training and to assess the

relevance of the training course to the learner’s roles. Indeed assessment might also look at the

content and presentation of the training event to evaluate its quality.

Outcomes Employers may want to measure the levels at which the learning has been transferred to the

workplace. This is easier where the training is concerned with hard and specific skills - this would

be the case for a train driver or signal operator but is harder for softer and less quantifiable

competencies including behavioural skills. If performance is expected to change as a result of

training, then the evaluation needs to establish the initial performance level of the learner.

In addition to evaluating the context, inputs, reactions and outcomes to training and development,

employers must continuously measure the costs. A cost / benefit analysis is usually conducted

prior to committing to any training initiatives. Costs must be monitored to ensure that they don't

scale over budget.

An alternative way to look at ROI is through the different levels of ROI analysis. This is explained

below by the Alkins UCLA model:

Alkins UCLA model

The UCLA Evaluation Model – Developed by Alkins, paralleled some aspects of the CIPP

model. (All but number 4 are alike) The Alkins model includes:

1. Systems assessment – Provide information about the state of the system.

2. Program planning – Assist in selection of particular programs to be effective in meeting

specific education needs.

3. Program implementation – To provide information about whether a program was introduced

to the appropriate group in the manner intended.


4. Program improvement – to provide information about how a program is functioning, whether

interim objectives are achieved, and whether unanticipated outcomes are appearing. (Not

similar to CIPP)

5. Program certification – To provide information about value of the program & it’s potential for

use elsewhere.

Alkins view of Evaluation – The process of ascertaining the decision areas of concern, selecting

appropriate information, and collecting and analyzing information in order to report summary data

useful to decision makers in selecting among alternatives.

1.4 Processes and principles for ROI

ROI is based on the following assumptions and principles – ROI needs to deliver on each of

these in order to be successful:

Cost effective to measure

Resource efficient to measure

Provides indicators of all 5 levels of evaluation

Is scalable and replicable across all learning classes, courses, curriculum and programs

Is benchmarkable for both internal and external comparisons

Uses reasonable assumptions based on industry proven principles and methodologies

Provides valuable business intelligence to a variety of stakeholders

Provides quantitative, financial evidence of return on investment

The key is deriving a monetized benefit from training. The benefit can be derived by calculating it

for a specific result such as sales, quality, productivity or cycle time. However, it may also be

derived by linking it to the known monetary value that is placed on human capital, an employee

salary. Let's use an example, if one buys a computer for R3,000 the expectation is that the

company will get R3,000 of value out of the computer. The computer may help a salesperson

increase sales or help a plant floor operator increase quality but the goal is to improve the user’s

job performance through the technology. The expectation is that at least R3,000 will be of benefit

in exchange for paying a cost of R3,000 to acquire the computer.

Compare this analysis to a person, (i.e. human capital). If the fully loaded salary (wages, benefits,

and overtime) of a newly hired employee is R50,000, the organization paying that expense

expects R50,000 of value from the employee. This value could come from their contributions in

one or more key business objectives such as sales, quality, productivity, cycle time, customer


satisfaction etc. But, in general, the organization expects a return of at least R50,000 from the

employee.

Now, say in our computer example our IT department added a R500 upgrade to it. The upgrade

is intended to make the machine faster, more resistant to bugs, and more accurate in its

processing computations. The business result is more productive employees, higher quality and

reduced cycle-time for a user of the computer. The expectation is that the R500 spent on the

upgrade will result in at least R500 returned in various benefits.

Compare this analysis with training. We use training to upgrade our people just as we add

components to a computer to upgrade technology. Training and organizational development are

proven tools to add knowledge and skills to our workforce. So, if an employee goes to a R1,000

training event over a week long period, the goal is that the employee will leverage the training to

help achieve various business results back on the job. Such results include increased sales,

quality, customer satisfaction, productivity etc. The expectation is that the R1,000 spent on the

training will result in at least R,000 returned in various benefits.

Important to consider as part of the process of ROI are the factors that might inhibit ROI:

Inhibitors to ROI measurement

When proposing to carry out an ROI exercise, it is worthwhile bearing in mind that in some cases,

there may be reluctance, in a training or HRD group, to commit wholeheartedly. There are a

number of reasons why this might be the case:

Fear of a negative ROI, and the implications this might entail

Reluctance to commit the necessary budget to carrying out the exercise

The fact that this process takes time, and only really demonstrates results some months,

or even longer, after the completion of the training

One of the keys to gaining commitment has to be to allay these fears and concerns, by

demonstrating a process that can be carried out, and by taking a realistic approach to the costs

and overheads, and the expectations of the client. In the right situations, the insights into the

effectiveness of the training can provide important strategic benefits to the company such as:

Measuring contribution of HRD

Setting priorities

Focusing on results

Altering management’s appreciation and perceptions of training


By carrying out some pre-training ROI analysis, it should be possible to demonstrate that the

there is a likelihood of a positive ROI on training, and thus allay some of the fear, as well as

encourage the investment in the training.

Important process guidelines for a successful ROI implementation:

For an ROI study to be successfully implemented, it should be designed with the following in

mind:

The ROI measurement must be simple

The ROI process must be economical to implement

The assumptions, methodology and techniques must be credible, logical, methodical and

practical

The ROI process must be theoretically sound, without being over-complex

The ROI process must account for other factors, which can influence the measured

outcomes after training

The ROI process must be appropriate in the context of other HRD programs

The ROI process must be flexible enough to be applied pre and post training

The ROI process must be applicable with all types of data collected

The ROI process must include the costs of the training and measurement program


Section 2: Applying ROI Methodology

2.1 ROI implementation methodology

The ROI Process is a comprehensive process that provides a scorecard of six measures.

These measures represent input from various sources during different time frames. The

measures include:

1. Reaction and satisfaction

2. Learning

3. Application and implementation

4. Business impact

5. Return on investment

6. Intangible benefits

In addition, the ROI Process utilizes at least one technique to isolate the effects of the program

from other influences. This comprehensive measurement system requires success with many

issues and must become a routine part of the learning development cycle.

There are good reasons why return on investment is such a hot topic. Although the viewpoints

and explanations may vary, some things are very clear. First, in most organizations education

and training budgets have continued to grow year after year (Training, 1999). As expenditures

grow, accountability becomes a more critical issue. A growing budget creates a larger target for

internal critics, often prompting the development of an ROI process.

Second, Total Quality Management and Continuous Process Improvement have brought

increased attention to measurement issues. Today organizations measure processes and

outputs that were not previously measured, monitored, and reported. This measurement focus

has placed increased pressure on the education and training function to develop measures of

program success.

Third, the proliferation of new hardware and software has created a need for accountability with

technology. For years, the implementation of new technology has escaped accountability.

Organizations have literally been willing to buy new technology, not knowing if the application

would actually generate an appropriate return on investment. Project sponsors have been

burned by inappropriate and improperly designed technology implementations. Today, that

situation is changing, particularly in education and training. Administrators and executives who

fund new projects are asking for a process to demonstrate accountability, with a commitment to

measure the return on investment after initial implementation. This situation has caused more


interest in accountability for any project utilizing technology, particularly in the education and

training field.

Fourth, restructuring and reengineering initiatives and the threat of outsourcing has caused

education and training executives to focus more directly on bottom-line issues. Many education

and training processes have been reengineered so that programs are more closely aligned with

business needs, and maximum efficiencies are required in the training cycle. These change

processes have brought increased attention to evaluation issues and have resulted in measuring

the contribution of specific programs.

Fifth, the business management mindset of many current education and training managers

causes them to place more emphasis on economic issues within the function. Today’s education

and training manager is more aware of bottom-line issues in the organization and more

knowledgeable of operational and financial concerns. This new “enlightened” manager often

takes a business approach to education and training and ROI is a part of the strategy.

Sixth, there has been a persistent trend of accountability in organizations all over the globe.

Every support function is attempting to show its worth by capturing the value that it adds to the

organization. From the accountability perspective, the education and training function should be

no different from the other functions. It must show its contribution to the organization.

Seventh, top executives are now demanding return on investment calculations from departments

and functions where they were not previously required. For years, training and education

managers convinced top executives that training cannot be measured, at least to the monetary

contribution level. Yet, many of the executives are now aware that it can and is being measured

in many organizations, thanks in part to articles in publications aimed at top executives (William

and Mary Business Review, 1995). Due to this increased awareness, top executives are

subsequently demanding the same accountability from their training and education functions. In

some extremes, these functions are being asked to show the return on investment or face

significant budget cuts (Gerber, 1994). Others are just being asked for results. The CEO for a

global telecommunications company recently described it this way: “For years we have evaluated

training with measures such as number of participants, number of programs, length of programs,

cost of programs, and content of programs. These are input focused measures. Now, we must

show what these programs are doing for our company and speak in terms that we can

understand. We need output focused measures.”(private interview) These no-nonsense

comments are being repeated throughout major organizations.

The Balanced Scorecard

As discussed, ROI is really value to your stakeholder and can mean different things for different

stakeholders. Merely positioning a financial metric to a training manager won't solve their


measurement needs. They need feedback on instructor performance, courseware quality etc.

Hence the need for an ROI scorecard that has a balanced set of metrics that provides indicators

on all 5 levels of learning, not just a financial ROI.

What should these measures be? A suggestion is to have a small set of measures that are

comprised of data gathered in a consistent manner on a continual basis. If done, a scorecard with

such metrics can be generated in a real time manner for any learning event or combination of

events you choose.

Below are the key performance components that comprise the scorecard.

Level 1 - Satisfaction

Level 2 - Learning Effectiveness

Level 3 - Job Impact

Time to Job Impact

Barriers to Use

Post Training Support

Level 4 Business Results

Job Performance Change

Business Drivers Impacted By Training

Level 5 Return on Investment

The benefits of this implementation methodology for ROI are:

Measure and Improve Job Impact

It allows you to streamline the learning evaluation process, measure training performance, and

ultimately, improve job impact.

Drive Superior Business Results

By having access to real-time learning and performance data, it provides organizations with the

ability to increase performance and drive superior business results.

Improve Return on Learning Investment


Because organizations they can't manage what they don't measure, it is important to establish the

right performance measures for all key investments. Learning is without a doubt one of the most

important investments any company will make.

The global economy and the rapid advancement of technology have made today's workforce

more mobile than ever before. Increased competition in a worldwide marketplace forced

companies to tighten their belts and find ways to value engineer everything, including learning. To

that end, today's world-class learning organizations are finding innovative ways to design and

deliver training better, faster and cheaper. These organizations are then monitoring the effects of

these changes through comprehensive measurement systems, and it helps these organizations

improve their Return on Learning Investment.

Industry Benchmark Comparisons

By leveraging ROI results, you can capture data on a wide array of learning interventions and

provide extensive reporting capabilities to the organization. Many value-added services using

such benchmarks for the organization to improve the performance of their learning operations.

Increase Shareholder Value

It helps organizations increase their shareholder value. By leveraging market leading models

such as the ROI Process, it provides thought leadership in the corporate learning industry.

Accelerate Adoption of E-Learning Programs

ROI captures learning performance data on many different learning modalities. Because learning

evaluation data is captured from online learning events in addition to traditional instructor-led

learning interventions, one can capture valuable data that helps corporations successfully adopt

and implement e-learning solutions.

Accountability on Training Dollars

Many corporate learning professionals have difficulties measuring their performance and

demonstrating value to senior management. Increasingly, corporate learning professionals are

being asked to justify budgets. ROI helps solve this problem by providing measurement data for

all training rands spent and helps training professionals determine what initiatives are working to

drive better business results.

Actionable Intelligence


ROI provides organizations with actionable intelligence. The data that is provided to

organizations, and the comprehensive ways in which we display this data provides organizations

with the ability to quickly gauge how effective learning is, and makes decisions accordingly.

2.2 Developing a Problem Tree for ROI

In order to really apply ROI methodology insightfully, you as the practitioner need to

understand the context of the organisation within which you are applying it, and the specific

context (or problem) for which the training intervention was designed / implemented. In other

words – what was the training supposed to change or solve or make better in the

organisational environment.

The Problem Tree method is a planning method based on needs; however it is not a

mechanical translation of problems into objectives. While going through the process, taking

the different steps, there is continuously room for opportunities, new ideas and contributions

from the involved parties. Problem Tree Analysis should be followed by actual project

planning, e.g. with the Logical Framework approach. Alongside, or interwoven with the steps

of Problem Analysis (at target group level) and project planning (for the target group), one

should analyse the capacity and intentions of stakeholders and the wider institutional context,

so that relevant and realistic choices can be made on who does what.

The problem tree is a visual problem-analysis tool that can be used to specify and investigate

the causes and effects of a problem and to highlight the relationships between them. It is a


tool for the identification and analysis of the relevant causes of the main problems, which will

later form the bases for formulating solutions and objectives for the strategy. A discussion of

the causes can help to identify the segments of the organization that are most affected and

who should be specifically interested in participating in activities aimed at removing the

causes of the problem. Remember that each cause of the problem is also a problem in its

own right.

As the name implies, this tool resembles a tree. The roots of the tree, in the lower part of the

drawing, metaphorically represent the causes of the main problem. The tree trunk at the

centre of the drawing represents the main problem and the tree branches, on the upper side

of the drawing, provide a visual representation of the effects of the main problem.

The whole purpose of the problem tree is to define the main problems present in the

organisation in order to analyse and prioritise their causes as the first step towards effective

sustainable solutions. Probably the most important tool to keep in mind throughout this

process is a single question or rather a single word: 'WHY?' It is amazing how this short word

can generate unexpected insights, which greatly help in developing an effective solution or

strategy. Never be afraid of asking or wondering why something is happening, even if it

seems obvious."

An example of a problem tree:


2.3 Developing a results chain for ROI

The Results Chain – planning for and measuring impact of training interventions

Training is implemented in order to achieve certain changes or results in the work process. The

impact of the training, when being applied back at work, is the REUSTS that we are looking to

measure in ROI. So to learn how to create and populate a results chain is important in the

process of planning and calculating the ROI of the training interventions that have been selected.

The Simple Results Chain

The results chain expresses the cause-and-effect relationship between a project and results. It’s

a way of representing the statement: “Projects cause results.”


http://twogoats.files.wordpress.com/2008/02/results-chain-simple.jpg%00%E5%A1%B9%EF%92%81%E1%B4%BB%E4%A1%BF%E2%B2%AF%E5%B6%82%E8%97%84%E6%8C%A7%00%00%EA%AE%A5

We use the arrow to show causality; that is, to show that

something causes something else. So whenever you see the

arrow in the results chain, you can insert the word “cause” or

“lead to.” If you prefer, you can express it using “if / then”: If we

do this project, then we’ll see these results.

The Expanded Results Chain

Of course, if you’ve seen or worked with RBM frameworks, you know that things get a bit busier

than the simple results chain. Each of the two elements –

projects and results – in the chain above gets expanded, giving

us different components to consider. Even so, the results chain

remains a way of expressing the concept that projects cause

results.

Projects: Inputs and Activities

To begin, we expand the “projects” side of the results chain into two components: inputs and

activities.

Inputs are the resources that we will need to carry out the planned activities. They include things

like people, money, goods, materials, infrastructure, and technology.

Activities are the things that we do. Some familiar project activities include digging wells,

conducting training session, distributing seeds and tools, forming savings groups, and setting up

clinics.

In the results chain, and in RBM, projects include inputs and activities. According to the basic

principles of this management approach, the projects cause the results that are important.

Results: Outputs, Outcomes, and Impact

Now we expand the second half of the results chain: the results side. Results are divided into

three big categories called outputs, outcomes, and impact.

Outputs are the immediate results of activities. For example, the output of digging wells would be

the number of functioning wells in a community. Or the output of a training session would be the

number of trained individuals.


http://twogoats.files.wordpress.com/2008/02/results-chain-expanded.jpg%00%E5%A1%B9%EF%92%81%E1%B4%BB%E4%A1%BF%E2%B2%AF%E5%B6%82%E8%97%84%E6%8C%A7%00%00%EA%AE%A5

Outcomes are medium-term results caused by outputs. Examples of outcomes might be

households with access to clean drinking water or the percentage of trainees who start a

business or find a job.

The impact is the long-term, broad societal change that the outcomes lead to. For example,

improved longevity or decreased malnutrition might be the impact of a project.

Putting it all together

Our results chain now reads something like this:

Inputs lead to activities, which cause outputs, which cause outcomes, which cause the impact.

Another way of reading it would be:

If we have the inputs, then we can do the activities;

if we do the activities, then we will achieve the outputs;

if we achieve the outputs, then we will achieve the outcomes; and

if we achieve the outcomes, then we will contribute to the impact.

Illustrated below are the basic components of a results chain and the indicator / measure

terminology that is used to measure impact through the results chain.


The basic components of a results chain:

The terminology used in a results chain:


2.4 Developing indicators for ROI

Indicators are measurable or tangible signs that something has been done or that something has

been achieved. In some studies, for example, an increased number of television aerials in a

community has been used as an indicator that the standard of living in that community has

improved. An indicator of community empowerment might be an increased frequency of

community members speaking at community meetings. If one were interested in the gender

impact of, for example, drilling a well in a village, then you could use “increased time for

involvement in development projects available to women” as an indicator. Common indicators for

something like overall health in a community are the infant / child / maternal mortality rate, the

birth rate, nutritional status and birth weights. You could also look at less direct indicators such

as the extent of immunisation, the extent of potable (drinkable) water available and so on.

Indicators are an essential part of a monitoring and evaluation system because they are what you

measure and / or monitor. Through the indicators you can ask and answer questions such as:

Who?

How many?

How often?

How much?

But you need to decide early on what your indicators are going to be so that you can begin

collecting the information immediately. You cannot use the number of television aerials in a

community as a sign of improved standard of living if you don’t know how many there were at the

beginning of the process.

Some people argue that the problem with measuring indicators is that other variables (or factors)

may have impacted on them as well. Within a project it is possible to identify other variables and

take them into account. It is also important to note that, if nothing is changing, if there is no

improvement in the measurement of the key indicators identified, then your strategy is not

working and needs to be rethought.

Developing indicators

Step 1: Identify the problem situation you are trying to address. The following might be problems:

Economic situation (unemployment, low incomes etc)

Social situation (housing, health, education etc)

Cultural or religious situation (not using traditional languages, low attendance at religious

services etc)

Political or organisational situation (ineffective local government, faction fighting etc)

There will be other situations as well.


Step 2: Develop a vision for how you would like the problem areas to be / look. This will give you

impact indicators.

What will tell you that the vision has been achieved? What signs will you see that you can

measure that will “prove” that the vision has been achieved? For example, if your vision was that

the people in your community would be healthy, then you can use health indicators to measure

how well you are doing. Has the infant mortality rate gone down? Do fewer women die during

child-birth? Has the HIV / AIDS infection rate been reduced? If you can answer “yes” to these

questions then progress is being made.

Step 3: Develop a process vision for how you want things to be achieved. This will give you

process indicators.

If, for example, you want success to be achieved through community efforts and participation,

then your process vision might include things like community health workers from the community

trained and offering a competent service used by all; community organises clean-up events on a

regular basis, and so on.

Step 4: Develop indicators for effectiveness.

For example, if you believe that you can increase the secondary school pass rate by upgrading

teachers, then you need indicators that show you have been effective in upgrading the teachers

e.g. evidence from a survey in the schools, compared with a baseline survey.

Step 5: Develop indicators for your efficiency targets.

Here you can set indicators such as: planned workshops are run within the stated timeframe,

costs for workshops are kept to a maximum of R 2000 per participant, no more than 160 hours in

total of staff time to be spent on organising a conference; no complaints about conference

organisation etc.

With this framework in place, you are in a position to monitor and evaluate efficiency,

effectiveness and impact

Some specific and popular ROI indicators include:

The main ROI indicators from a finance perspective include the following:

Benefit to Cost Ratio

ROI Percentage

Payback Period

The benefit to cost ratio is probably the most relevant of the three. It is simply the monetized

benefit divided by the costs of the training. The costs should also be fully loaded for


conservatism. Typical costs need to include cost items such as needs assessment, design,

delivery, materials, overhead, evaluation, lost work time of participants and travel expenses of

participants. The benefit to cost ratio will then be a conservative view on the financial

ramifications of your training program. Ratios greater than 1 are positive in ROI. Ratios less than

1 are negative in ROI and ratios equal to one are break even. So, for example, if you have a

benefit to cost ratio of 2.5 that means the training program returned 2.5 dollars for every dollar

spent on it.

Another financial ratio is the ROI percentage. This is the benefit less the cost divided by the cost,

expressed as a percentage. Although more common than benefit to cost ratio, the benefit to cost

ratio is a more typical measure in training's use of ROI financial measures because it is not as

hard to interpret as the ROI percentage and has less tendency to be compared to other ROI

projects that are not human capital based.

Finally is the payback period. This is a time based financial metric. It tells you how many months

(or whatever time period you use) are required before you break even on the investment, after

which is a positive return. It is good to provide time -based metrics to balance out your scorecard.

2.5 Developing a measuring framework for ROI

Assessment of training may be conducted at 5 different levels, as illustrated in the matrix in

previous sections. These include (1) Evaluating the reaction of participants, (2) Measuring the

learning that occurred, (3) Assessing the on the job behavior, (4) Identifying business results of

training, and (5) Calculating the return on investment (ROI).

A comprehensive evaluation of ROI only occurs at Level 5. Most organizations compute the value

of training using data collected at the other four levels. These computations are based on a

number of assumptions and estimates; the lower the level of evaluation, the greater the use of

estimates and assumptions. For instance, at Level 1, a trainee's action plan might indicate that he

/ she estimates a 10% increase in productivity as a result of the training. This productivity gain

could be factored for with a confidence rating and then translated into increased revenue or profit.

A thorough ROI analysis is typically conducted for only 10-20% of all training programs. The

collection and analysis of data can be time consuming and expensive. Computing the ROI of

training at Level 5 involves these four steps: (1) isolating the effects of training, (2) converting

these effects (benefits) into monetary values, (3) calculating the costs of the training, and (4)

comparing the value of the effects to the incurred costs.


(1) Isolating the effects of training

To determine the ROI of training you must be able to measure the changes that occur as a result

of training. Consequently, you must know what the performance or level of knowledge was before

you began the training initiative. Pre-training data measurement might include frequency of

errors, labor hours per unit of production or service, dollars of scrap materials, number of returned

or defective products, volume of lost sales, absentee rates, turnover rates, or survey ratings

indicating customer dissatisfaction. Unfortunately, many companies do not maintain detailed

records, and therefore, the pre-training data is not available. Delaying training in order to

accumulate pre-training data may not be a wise decision.

An option to pre-training and post-training data comparisons is to use two sample groups of

employees. One test group receives the training while the control group does not. Productivity or

performance of the two groups is measured and compared. The challenge for HR managers is to

make sure that the two groups are similar except for the training variable. This requires

assessment and equal distribution of the skills in each group as well as a careful matching for

age, race, gender, educational level, years of experience, previous training, and other variables.

Isolating the effects of training requires identification of all of the key factors that impact employee

performance and business outcomes. Focus groups, questionnaires, surveys, and observations

will facilitate collection of this data. Possible contributors include employees, senior managers,

supervisors, customers, vendors, and training and human resource development specialists.

(2) Converting the effects of training into monetary values

The effects, or benefits, of a training program should always be identified, qualified and converted

to dollars with input from management. Trainees' supervisors, department or division heads,

senior level executives or even the board of directors are in excellent positions to observe

changes in performance or impact on the bottom line. Their decisions and data may be far more

objective and credible than if the HR manager makes all of the decisions regarding the scope,

impact and duration of training benefits. Costs are known up front. Benefits may accrue slowly

over time. Accurately estimating the number of times a course will be used, the number of

employees that will be affected, the dollar impact of changes in quantity or quality, and the extent

to which training will affect results requires skill, insight, and clearly defined objectives.

Effects can be tangible or intangible and are frequently referred to as "hard data" and "soft data."

Hard data is quantitative, statistical, number oriented and easily translated into monetary benefits.

Soft data is qualitative and refers to intangible benefits that are subjective and thus are more

difficult to measure and translate into monetary benefits.


Samples of both kinds of data are listed below.

Sample "hard" data for determining the effects of training data

Productivity measures (quantity or market value)

Quality measures (number of rejects or cost of rejects)

Materials costs (amount per unit of production or amount of waste or scrap)

Labor hours per unit of production

Labor costs per unit of production

Hours of "down time" due to equipment failure, etc

Absenteeism and tardiness rates

Turnover rate

Workers compensation claims - nature and number of injuries or illnesses, days of lost

work or "light duty" work

Number of grievances / legal claims / lawsuits

Time required to fill vacant positions

Time required to fill an order, respond to a telephone call, resolve a complaint, etc.

Number of sales or dollar value of sales per customer

Percent of market share

Customer satisfaction rating or index

Number of repeat customers

Number of accounts or dollar value of accounts more than 30, 60, 90 days past due

Sample "soft data" effects or benefits of training

Improved job satisfaction

Improved teamwork

Increased organizational commitment

Improved succession planning

Increased communication regarding career paths

More clearly defined promotion opportunities

(3) Calculating the cost of training


HR managers generally are able to establish the costs of a training program. When calculating

costs, remember to carefully consider all indirect costs, such as staff time, use of existing

materials, equipment, classrooms etc. The term "fully loaded costs" is sometimes used to

designate that the costs of a program include both direct and indirect costs.

Sample training costs:

Course Development - needs analysis, design, writing, illustrating, validating tests and

evaluation instruments

Wages and Salaries of HR staff, managers and employees involved in course design and

development

Wages and Salaries of instructor and any onsite support staff

Wages and Salaries of trainees (sometimes referred to as "seat time")

Wages for temporary or contract workers hired to maintain productivity / service during

regular employees' training time

Loss of revenue while trainees are involved in training activities

Payments to outside trainers, outside training companies, or consultants

Instructional Materials - printing, copying, or reproducing manuals, videos, computer

disks, and flip charts; pens, pencils, paper, etc.; and / or purchase of materials from

outside vendor

Equipment and Facilities - including all types of AV equipment, computer hardware and

software, classroom overhead or rental of training facilities or equipment

Administration - marketing, scheduling, registration, testing, documentation, copying,

collating, long distance phone calls, postage, (administrative staff time and costs

associated with administrative duties)

Logistics - lodging, meals, tips, refreshment breaks, and shipping costs

Travel time - to and from training

Tuition reimbursement

(4) Comparing the value of effects to the incurred cost

It may not always be feasible to measure ROI. It is doubtful that companies are setting up web

sites and developing e-commerce initiatives after a careful and exhaustive analysis of their ROI.

These organizations realize that rapid changes are occurring, and they will lose future business if

they fail to adapt business methods to new technologies.


Similarly, if HR becomes focused on getting every cost down to a line item, and then comparing

the costs with the benefits, every line item could be debated. Are you able to clearly show that it

was the training that made the difference in a customer's satisfaction? Are you sure you will

provide training for (x) number of employees before this training program becomes obsolete? Are

you sure that the training is responsible for the improvement in employee performance rather

than a new manager or a change in the compensation plan? When did you decide to measure

training results, in 3 months or 6 months? What time frame is logical or reasonable for your

industry? What would happen to your ROI calculations if turnover increased or decreased?

Sometimes you may need to focus on the big picture, the changes that are occurring, rather than

a line-by-line item of cost benefit analysis or ROI.

New employee orientation and diversity training are examples of types of training where it may be

difficult to measure ROI. New employees have varied backgrounds. For some employees, this is

their first job, and they might find it difficult to absorb all of the information given to them. Others

employees will have years of work experience and have attended several new employee

orientation programs. They already have a base of knowledge and are looking for the kind of

information that is specific to your company. Diversity training may also be an initial experience or

a review. It may be very difficult to hold all variables constant when you are trying to look at the

effect of training. However, few would argue that new employee orientation or diversity training

should be eliminated simply because you cannot easily demonstrate a ROI. Orientation programs

set the tone for the entire employer / employee relationship. Turnover rates or productivity

measurements of new employees using a test group and a control group could provide data for

estimating the value of orientation programs if implemented throughout the entire organization.


Section 3: Data Collection and Analysis for ROI

3.1 Basic research design and methodology for ROI

It is necessary to collect data at various levels to build up a proper picture of the influence of

learning (as per the Kirkpatrick model, or any other model). In order to determine which data

items should be collected, and when, you need to agree with the customer your approach to the

questions as suggested below. These questions describe some of the potential areas where

returns can be achieved; it is down to you and the client to decide exactly which of these factors

apply in any given situation.

Beware that, sometimes, benefits can be hidden; for example, consider the business benefits of

“certification for regulatory compliance” in the ROI calculation; it may be that compliance is seen

as a pure overhead cost of doing business; but in this case, training can reduce the cost of

compliance (and the costs of non-compliance), and provide benefits in enabling new business

activities, or better quality of customer service resulting from the compliance-training etc.

Examples of such factors may be

How can you quantify “competitive advantage” resulting from a more highly trained and

competent workforce, resulting in shareholder value?

o Competitive advantage is probably seen differently by every organisation; what

constitutes competitive advantage may perhaps be better seen as a compilation

of factors, such as quality, cost, customer satisfaction; but there may be

intangible elements such as “company image in the market” which amounts to

more than the sum of the individual components of competitive advantage.

How can you quantify the value of improvement in managers’ long term decision making

resulting from improved techniques in situational analysis and decision making?

o This may require some quite sophisticated analysis; maybe this can be

determined by offering simulation exercises, or role / game playing, and deriving

a score from that.

Note: you should try to estimate beforehand what a reasonable ROI is. Carrying out an expensive

exercise to show a 300% ROI is great; unless comparable training exercises in the clients

industry for equivalent training typically show 750%! Or even better, how does the client typically

view 300%? Is this good in their circumstances, or is it lower than their typical ROI as determined

from previous training?


3.2 Data collection options for ROI

It is important to note that it is critical to gather data from participants of ROI at least two points in

time. The Post Event instrument will gather data immediately after a learning intervention. It is at

this point that the participant estimates via forecasting their job performance, isolation and

confidence ratings. This is important because a huge value in ROI is to make it a predictive tool

not a reactive or historic tool. Forecasting ROI prior to the passage of time can be a very valuable

tool to make business decisions just as sales forecasts drive sales decisions and accounting

forecasts drive accounting decisions.

The Follow-Up is a second exercise to re-collect data when the participant is back on the job and

time has passed. Here the data on job performance, isolation, and adjustment is no longer a

prediction but a realistic estimate of what has really occurred. This is critical to do to understand

reality. Just as sales people review actual versus forecast so to should training personnel view

Post Event vs. Follow Up

Sources of data can include:

Organisational Performance Records, showing outputs and measurements taken as part

of the business’ normal reporting process

Testing and certification assessment records

Participant feedback

Instructor feedback

Feedback from participants’ supervisors / managers

Feedback from participants’ subordinates

Team / group peer feedback

Feedback from other internal or external groups (e.g. HR training departments)

Data Collection methods

There are various methods of collecting data; some of these, such as surveys and tests can be

automated, to reduce the time and resource required to carry out the data collection:

Questionnaires; Knowledge tests; certification programme:

To capture subjective or objective information about participants’ performance changes

Comparison of pre-training and post-training assessments (“compare apples with

apples”)

Surveys


To capture subjective feedback on attitudes, beliefs and opinions

Comparison of pre-training and post-training assessments (“compare apples with

apples”)

On the job Observation

To capture objective data on performance in carrying out specific task(s) in the normal

work environment

Interviews

To capture in depth subjective or objective information about participants’ performance

changes, and / or to capture subjective feedback on attitudes, beliefs and opinions

Focus groups

An extension to the Interview technique, involving in depth discussion as well as

individual feedback; especially valuable for subjective quality judgments covering a group

of participants

Action plans (or Performance contracts) and Program assignments

To measure the actual performance of a participant in carrying out a pre-determined

task(s), or to assess the participant’s understanding of what actions would be taken in

performing the task(s)

Performance data monitoring

To capture specific performance of the participants against measured performance

criteria

3.3 Data analysis options for ROI

Following the planning process, implementation begins. Data collection and analysis is central

to the ROI process. Both hard data, representing output, quality, costs, and time; and soft

data, including work habits, work climate, and attitudes, are collected. Data are usually

collected during two time frames. During the training process, Level 1 and Level 2 data are

collected. Following the training, Level 3 and Level 4 data are collected.

A variety of methods are used to collect the post-program data to be used in the ROI analysis

and evaluation. Important to note here is that he method selected to collect the data, will in

turn affect the way that they data can be analysed, compared and integrated into the ROI

calculation – so selection is very important:


Follow-up surveys are taken to determine the degree to which participants have utilized

various aspects of the program. Survey responses are often developed on a sliding

scale and usually represent attitudinal data. Surveys are useful in collecting Level 3 data.

Follow-up questionnaires are administered to uncover specific applications of education

and training. Participants provide responses to a variety of open-ended and forced

response questions. Questionnaires can be used to capture both Level 3 and 4 data.

On-the-job observation captures actual skill application and use. Observations are

particularly useful in customer-service training and are more effective when the observer

is either invisible or transparent. Observations are appropriate for Level 3 data.

Post-program interviews are conducted with participants to determine the extent to

which learning has been utilized on-the-job. Interviews allow for probing to uncover

specific applications and are appropriate with Level 3 data.

Focus groups are conducted to determine the degree to which a group of participants

have applied the training to job situations. Focus groups are appropriate with Level 3

data.

Program assignments are useful for simple short-term projects. Participants complete

the assignment on the job, utilizing skills or knowledge learned in the program.

Completed assignments can often contain both Level 3 or 4 data.

Action plans are developed in programs and are implemented on the job after the

program is completed. A follow-up of the plans provide evidence of program success.

Level 3 and 4 data can be collected with action plans.

Performance contracts are developed where the participant, the participant’s

supervisor, and the instructor all agree on specific outcomes from education and training.

Performance contracts are appropriate for both Level 3 and 4 data.

Programs are designed with a follow-up session which is utilized to capture evaluation

data as well as present additional learning material. In the follow-up session, participants

discuss their successes with the program. Follow-up sessions are appropriate for both

Level 3 or 4 data.

Performance monitoring is useful where various performance records and operational

data are examined for improvement. This method is particularly useful for Level 4 data.

The important challenge is to select the data collection method or methods appropriate for the

setting and the specific program, within the time and budget constraints of the organization.

Isolating the Effects of the Program

An often overlooked issue in most ROI evaluations is the process to isolate the effects of

education and training. In this step of the ROI process, specific strategies are explored that

determine the amount of output performance directly related to the program. This step is

essential because there are many factors that will influence performance data after education and


training programs are conducted. The specific techniques utilized at this step will pinpoint the

amount of improvement directly related to the program. The result is increased accuracy and

credibility of the ROI calculation. The following techniques have been utilized by organizations to

address this important issue:

A control group arrangement may be used to isolate impact. With this technique, one

group participates in the program while another, similar, group does not. The difference

in the performance of the two groups is attributed to the program. When properly setup

and implemented, the control group arrangement is the most effective way to isolate the

effects of education and training.

Trend lines are used to project the value of specific output variables, if the program had

not been undertaken. The projection is compared to the actual data after the program

and the difference represents the estimate of the impact. Under certain conditions this

strategy can be an accurate way to isolate the impact of education and training.

When mathematical relationships between input and output variables are known, a

forecasting model is used to isolate the effects of a program. With this approach, the

output variable is predicted using the forecasting model with the assumption that the

program is not conducted. The actual performance of the variable after the program is

then compared with the forecasted value to estimate the impact of education and training.

Participants estimate the amount of improvement related to education and training.

With this approach, participants are provided with the total amount of improvement, on a

pre-and post-program basis, and are asked to indicate the percent of the improvement

that is actually related to the program.

Supervisors of participants estimate the impact of education and training on the output

variables. With this approach, supervisors of participants are presented with the total

amount of improvement and are asked to indicate the percent related to the program.

Senior managers estimate the impact of education and training. In these cases,

managers provide an estimate or “adjustment” to reflect the portion of the improvement

related to the program. While perhaps inaccurate, there are some advantages of having

senior management involved in this process, such as senior management ownership of

the program.

Experts provide estimates of the impact of education and training on the performance

variable. Because the estimates are based on previous experience, the experts must be

familiar with the type of training and the specific situation.

In supervisory and management training, the subordinates of participants identify changes in the work climate which could influence the output variables. With this

approach, the subordinates of the supervisors receiving training determine if other

variables changed in the work climate that could have influenced output performance.


When feasible, other influencing factors are identified and the impact estimated or calculated leaving the remaining unexplained improvement attributed to education and

training. In this case, the influence of all of the other factors are developed and the

program remains the one variable not accounted for in the analysis. The unexplained

portion of the output is then attributed to the program.

In some situations, customers provide input on the extent to which training has influenced their decision to use a product or service. Although this strategy has limited

applications, it can be quite useful in customer service and sales training.

Collectively, these ten techniques provide a comprehensive set of tools to isolate the effects of

education and training.

3.4 Evaluation techniques for ROI

Evaluation is different to monitoring and collecting data for the ROI analysis process. Evaluation

is understood as the end result of a process of action – this evaluation can take place at different

levels. Here we refer again to Kirckpatrick’s four levels and use his model to illustrate the

evaluation techniques that can be used at each level of intervention:

Kirkpatrick's four levels of training evaluation in detail

This grid illustrates the Kirkpatrick's structure detail, and particularly the modern-day interpretation

of the Kirkpatrick learning evaluation model, usage, implications, and examples of tools and

methods. This diagram is the same format as the one above but with more detail and explanation:

Evaluation level and

type

Evaluation description and characteristics

Examples of evaluation tools and methods

Relevance and practicability

1. Reaction Reaction evaluation is how the delegates felt, and their personal reactions to the training or learning experience, for example:

Did the trainees like and enjoy the training?

Did they consider the training relevant?

Was it a good use of their time?

Did they like the venue, the style, timing, domestics, etc?

Typically 'happy sheets'.

Feedback forms based on subjective personal reaction to the training experience.

Verbal reaction which can be noted and analysed.

Post-training surveys or questionnaires.

Online evaluation or grading by delegates.

Subsequent verbal or written reports given by delegates to managers back at their jobs.

Can be done immediately the training ends.

Very easy to obtain reaction feedback

Feedback is not expensive to gather or to analyse for groups.

Important to know that people were not upset or disappointed.

Important that people give a positive impression when relating their experience


Level of participation.

Ease and comfort of experience.

Level of effort required to make the most of the learning.

Perceived practicability and potential for applying the learning.

to others who might be deciding whether to experience same.

2. Learning Learning evaluation is the measurement of the increase in knowledge or intellectual capability from before to after the learning experience:

Did the trainees learn what was intended to be taught?

Did the trainee experience what was intended for them to experience?

What is the extent of advancement or change in the trainees after the training, in the direction or area that was intended?

Typically assessments or tests before and after the training.

Interview or observation can be used before and after although this is time-consuming and can be inconsistent.

Methods of assessment need to be closely related to the aims of the learning.

Measurement and analysis is possible and easy on a group scale.

Reliable, clear scoring and measurements need to be established, so as to limit the risk of inconsistent assessment.

Hard-copy, electronic, online or interview style assessments are all possible.

Relatively simple to set up, but more investment and thought required than reaction evaluation.

Highly relevant and clear-cut for certain training such as quantifiable or technical skills.

Less easy for more complex learning such as attitudinal development, which is famously difficult to assess.

Cost escalates if systems are poorly designed, which increases work required to measure and analyse.

3. Behaviour

Behaviour evaluation is the extent to which the trainees applied the learning and changed their behaviour, and this can be immediately and several months after the training, depending on the situation:

Did the trainees put their learning into effect when back on the job?

Were the relevant skills and knowledge used

Was there noticeable and

Observation and interview over time are required to assess change, relevance of change, and sustainability of change.

Arbitrary snapshot assessments are not reliable because people change in different ways at different times.

Assessments need to be subtle and ongoing, and then transferred to a suitable analysis tool.

Assessments need to be

Measurement of behaviour change is less easy to quantify and interpret than reaction and learning evaluation.

Simple quick response systems unlikely to be adequate.

Cooperation and skill of observers, typically line-managers, are important factors, and difficult to control.

Management and


measurable change in the activity and performance of the trainees when back in their roles?

Was the change in behaviour and new level of knowledge sustained?

Would the trainee be able to transfer their learning to another person?

Is the trainee aware of their change in behaviour, knowledge, skill level?

designed to reduce subjective judgement of the observer or interviewer, which is a variable factor that can affect reliability and consistency of measurements.

The opinion of the trainee, which is a relevant indicator, is also subjective and unreliable, and so needs to be measured in a consistent defined way.

360-degree feedback is useful method and need not be used before training, because respondents can make a judgement as to change after training, and this can be analysed for groups of respondents and trainees.

Assessments can be designed around relevant performance scenarios, and specific key performance indicators or criteria.

Online and electronic assessments are more difficult to incorporate - assessments tend to be more successful when integrated within existing management and coaching protocols.

Self-assessment can be useful, using carefully designed criteria and measurements.

analysis of ongoing subtle assessments are difficult, and virtually impossible without a well-designed system from the beginning.

Evaluation of implementation and application is an extremely important assessment - there is little point in a good reaction and good increase in capability if nothing changes back in the job, therefore evaluation in this area is vital, albeit challenging.

Behaviour change evaluation is possible given good support and involvement from line managers or trainees, so it is helpful to involve them from the start, and to identify benefits for them, which links to the level 4 evaluation below.

4. Results Results evaluation is the effect on the business or environment resulting from the improved performance of the trainee - it is the acid test.

Measures would typically be business or organisational key performance indicators,

It is possible that many of these measures are already in place via normal management systems and reporting.

The challenge is to identify which and how relate to the trainee's input and influence.

Therefore it is important to identify and agree accountability and

Individually, results evaluation is not particularly difficult; across an entire organisation it becomes very much more challenging, not least because of the reliance on line-management, and the frequency and scale of changing structures, responsibilities and


such as:

Volumes, values, percentages, timescales, return on investment, and other quantifiable aspects of organisational performance, for instance; numbers of complaints, staff turnover, attrition, failures, wastage, non-compliance, quality ratings, achievement of standards and accreditations, growth, retention, etc.

relevance with the trainee at the start of the training, so they understand what is to be measured.

This process overlays normal good management practice - it simply needs linking to the training input.

Failure to link to training input type and timing will greatly reduce the ease by which results can be attributed to the training.

For senior people particularly, annual appraisals and ongoing agreement of key business objectives are integral to measuring business results derived from training.

roles, which complicates the process of attributing clear accountability.

Also, external factors greatly affect organisational and business performance, which cloud the true cause of good or poor results.


Section 4: ROI Calculations

4.1 Various approaches to calculating ROI

ROI is a key financial metric of the value of training investments and costs. It is a ratio of net

benefits to costs, expressed as a percentage. The formula can be expressed as:

[(monetary benefits – cost of the training) / cost of the training] x 100

If the ROI from a training programme is calculated at 335, it means that for every rand spent,

there has been a return of R3,55 in net benefit, after all costs are factored in. The exact form of

that benefit depends on the objectives of the learning programme, i.e. better telephone technique

application and communication skills after training. The total cost of the employee to the

organisation is calculated including floor space, p.c. rental, etc. If the employee answers 3,000

calls per month divided into the cost of R7, 000. To the organisation, the cost would be R2.33 per

call cost to the organisation. After training the employee answers 3,180 calls per month. The cost

of the employee to the organisation is now R2.20. The value of the training programme is then

calculated as per the above formula to determine the monetary value of the benefit

Some factors to take into account when calculating the returns on the investment

Return (Benefit) factors

Reaction and plan

o What is the immediate reaction as to whether the training / learning has been, or

is likely to be, useful

o How that will translate into immediate / short-term identification of how the

learning will be applied to the job

o What are the immediate expectations of how the learning will positively affect the

individual’s performance

Measuring Learning


o How has the learning affected the individual’s skills and knowledge level required

to perform effectively; what is the expected change in performance levels and

productivity?

o Has the learning positively or negatively affected the individual’s attitude to their

work?

Assessing application to the job

o How has the training actually changed the individual’s previous job performance

against measured performance standards and criteria?

o Has the individual developed an ability to perform new tasks?

o Has the individual been able to save costs which would have been incurred

without the training?

Business results (benefits and cost savings)

o What is the actual numerical change in the measured metrics for the specified

data items?

o What are the actual monetary values of the changes achieved?

o What other identifiable cost savings have resulted from the training methodology

used, compared to alternative methodologies (e.g. costs saved by using

elearning vs. classroom)?

Investment (Cost) factors

Some of the factors to take into consideration when calculating the total cost of a training

programme or course

Money spent on the training / learning

o Equipment upgrades (e.g. for an advanced e-learning package requiring

additional hardware and software, servers etc, or for additional equipment in a

classroom) ***

o External content development and maintenance resource (for training and testing

/ certification

o Classroom costs (space costs, equipment rental, catering, materials, promotional

items, stationery)

o Cost of infrastructure overheads for elearning and virtual classroom facilities (a

share of the ongoing costs of having access to these facilities) ***

o Communications costs to reach internal and external audiences

promotional programmes

mailers and flyers

poster design and production

conference calls and meetings

PR


o Costs incurred in the evaluation exercise

Time and effort used

o Who

Training Project team

Training programme managers

Instructors

Senior management time and attention

Training administration, records administration

Learners / students

Learners’ managers (e.g. time spent contributing to the evaluation)

o What activities

Training Needs Analysis and Programme design

Materials development and testing

Accreditation design and development

Communications and training

Evaluation and ROI calculation

Both of these items are amenable to financial cost calculation. Note: Time and effort should take

into account both:

Actual salary and expenses of staff

Opportunity cost of not performing other tasks

*** Note, some items of infrastructure costs, such as implementations of Learning Management

Systems, may be so large as to make for great difficulty in allocating costs to a specific

programme. In these cases you should decide whether to:

allocate a “nominal share” of shared facilities or infrastructure cost to any particular

programme of learning

calculate an actual cost of shared facilities or infrastructure cost where it has been in

operation for long enough to make a reasonable calculation

allocate the full cost of this infrastructure, if it is implemented purely for the purpose of

supporting the project

In all these cases, the basis of calculation should be clearly stated and understood.


Steps for calculating ROI

Step 1: Create an ROI Measurement Plan

To achieve the ROI measurement requires that the ROI process is planned early, and that two

planning documents are prepared:

The Data Collection Plan

State the objectives of the training / learning

State the objectives of each phase of data collection at each evaluation Level

Identify any previously used metrics, values or methodologies used by the

client, and determine their suitability for the current exercise

Select the appropriate evaluation methods

Identify the audiences who will be surveyed for data

i. The learner

ii. The manager (focused on levels 3 and 4)

iii. The analyst (for analytically determined performance-change data based

on available performance measurement data)

Set the timing for the data collection

i. Pre-training data collection (benchmarking the current situation)

ii. Post-training data collection (comparative data (“apples to apples

comparison”))

1. immediately post-training (initial reaction and assessment)

focused on levels 1 and 2

2. at a later date when the effect of the training has had time to

make itself felt in the learner’s job performance – focused on

levels 3 and 4

Allocate responsibilities for data collection and analysis

The ROI Analysis Plan (This is a continuation of the Data Collection Plan, capturing

information on the key items needed to develop the actual ROI calculation.)

List Significant Data items (usually Level 4 or 3) to be collected

i. Benefit Factors

ii. Cost Factors

Methods to isolate effects of the learning / training from other influences

Methods to convert data to numerical values

Intangible benefits

Other influences

Communication targets


Step 2: Collect Data

As described above, data will have to be collected at various points in the training process, to be

able to carry out effective ROI calculations. The following notes may be helpful in planning your

data collection exercise.

1. Identify the purposes of the evaluation. State clearly what the evaluations are to

measure and what the goals of the training are intended to be. Be as specific as

possible about the goals of the training, and make sure these goals address the

performance enhancement, business improvement or cost savings expectations.

2. Select the evaluation instruments and methodology. Identify how the data will be

collected and analysed (see below for different methods of data collection)

3. Establish the timing for the data collection. Decide whether pre-training analysis is

required, or post training analysis, or both. (E.g. pre-training and multiple post-

training assessments may be necessary to effectively identify the skills changes in

Levels 2, 3 and 4.)

4. Carry out the data collection at the levels 1-4 indicated above

Step 3: Isolate the effects of training

Once the training program is complete, it is essential to identify which performance improvements

have resulted from the training, and which improvements are co-incidental and may not be

directly relevant to the training.

Tools to isolate the effects of training can include:

Control Groups; comparing the change in performance of a group which has undertaken

training, to the performance change of a similar, untrained group, can provide a factoring

value by which the total measured change can be adjusted.

Trend lines; these show the trends in performance change, which would have been

expected if the training had not taken place; these can be compared to actual

improvement measurements. Draw a trend line, derived from previous performance data,

and extend into the future; then compare this later with the actual data derived from the

results of post-training evaluation.

Mathematical forecasting. This may be appropriate where there are several factors to

consider in the change in performance (e.g. sales increase due to training, and an

increase in marketing expenditure). It may be possible to calculate the expected trend

due to the marketing intervention and calculate the difference as being due to the training

(see Phillips “ROI in Training and Performance Improvement Programs”).

Participants’ estimates of the impact.

Supervisors’ or managers’ assessments of the impact of training on measured

performance improvement.


Senior management estimates of the overall value of training programs on the business,

taking into account other factors they are aware of.

Having isolated the effects of training, it may be necessary to adjust the numeric results of the

data collection exercise to reflect the fact that estimates may be “inflated”; on average, individuals

may believe that training has resulted in a 25% performance improvement; in practice it may be

less than that, as the group concerned may have a tendency to over-estimate the effects of the

training. Taking a conservative approach, and applying an adjusting factor, may help to reduce

bias in the survey data.

Step 4: Convert data to monetary value

In this stage it is important to estimate the financial value of the various changes resulting from

the training, and to identify the total costs incurred in implementing the training program.

There are various strategies for estimating the value of performance changes:

Output data converted to profit contribution or cost savings

o Direct costs saved

o Increased volumes of output produced

o Timeliness of output

Cost of quality calculated and quality improvements converted to cost savings or

increased profitability

Cost savings (salaries and overheads) in reductions in participants’ time in completing

projects.

Internal or external experts may be able to estimate the values of the performance

improvements gained.

Participants or their supervisors / mangers can estimate the cost savings or value of

increased productivity

Having calculated the direct financial value of the performance enhancements, it is also

necessary, wherever possible, to estimate the value of the more “intangible benefits”, such as:

Increased job satisfaction, and the benefits of increased staff retention and reduced

recruitment costs

Increased organisational commitment

Improved teamwork

Improved customer service

Reduced problems and complaints

Reduced conflicts


To calculate the cost of the training program, ensure you include:

Cost of external training services purchased

Cost of training materials supplied

Cost of internal training staff involvement

Cost of facilities used for the program

Travel, accommodation and other incidental costs

Salaries and benefits of the participants

Administrative and overhead costs of the training function.

The costs of carrying out the ROI on the training program.

Step 5: Calculate the ROI

ROI can be expressed in 3 different ways

1. Benefit / Cost Ratio

BCR = Program Benefits

Program Costs

BCR uses the total benefits and the total costs, and are expressed as a ratio (e.g. 10:1)

For example:

If you gain a benefit of $1M in 12 months, and the cost of training is $250K for the same

period, the BCR is 4:1

2. ROI %

ROI = Net Program Benefits

Program Costs x 100

In ROI, the costs are subtracted from the total benefits to produce net benefits, which are

then divided by the costs. This shows the net benefits to the company after the costs are

covered.


For example:


period, the net benefit is $750K; divided by the costs of $250K (and multiplied by 100) this

equates to a 300% ROI

3. Break-even time

Break-even time in months = Investment

Benefit x Period in months

For example:


period, the break-even time is $250k divided by $1M, times 12 months = 3 months

It is possible for business benefits to last more than 1 year, but in most cases, the effect of

training is more difficult to assess over longer periods, so typically 1 year benefits are calculated.

Longer term programs can be measured over multiple years.

Calculating ROI requires that business results data must be converted to monetary benefits. It is

important to be able to allocate financial value to results such as

Improved productivity

o Time saved

o Output increased

Enhanced quality

Reduced employee turnover

Decreased absenteeism

Improved customer satisfaction

ROI calculations can be used in 2 ways;

To identify the returns on a specific training programme, and compare these to expected

results

To examine alternative approaches, and their respective expected costs and benefits,

e.g.:

o blended vs. purely classroom

o multiple course programme vs. single course


o limited scope vs. extended scope (e.g. addressing a few goals, or addressing a

larger number of goals)

The first use is a relatively straight forward analysis of costs and benefits, using the models and

tools provided.

The second use however can also be accomplished using the tools. In this case, you can use the

tools to provide a number of different views of the ROI data relating to the different approaches,

and compare the results by applying the estimated data to the final ROI calculations.

4.2 Tools and templates for calculating ROI

Phillips guiding principles include elements of what he refers to as estimation, isolation and

adjustment. These are the cornerstones to monetizing a benefit (the numerator in our ROI

equation) and linking it to training.

Estimation is a process commonly used in business today. Sales people will estimate their future

sales, accounting people will estimate the cost of a warranty or claim that is expected in the

future. So to can training personnel ask that participants estimate the job performance impact that

a training program will have on their job. Participant estimation, as it is commonly referred, is not

estimating the performance solely related to training but asks participants to estimate job

performance changes in general, including among other factors, training.

For example, if one attends sales training, one might estimate an increase in job performance but

that increase could be related to other factors such as a competitor going out of business that

increases sales performance more so than training. So, estimates of performance change need

to take into account many factors, not just training. Those factors include process changes,

people changes, marketplace changes, technology changes and of course training.

When estimating the increase, the participant should think carefully about all the factors

mentioned. They may want to review historic data and forecast data to reasonably factor into their

overall performance change.

Logically, the training department is keenly interested in the effect training had on the

performance improvement. So, the next step is to isolate the estimated increase in performance

to just training. In this part of the process, the participant should estimate the how much the

training has or will influence job performance, relative to the other factors and assign a value to it.

So if the sales person felt that training was the strongest factor that caused change or will be the

driving force behind future change it would receive a higher value than not.


Finally, because participant estimation and isolation is participant driven one must adjust any

resulting ROI calculation for the estimate. Again, in other facets of business this is commonly

done. Using analysis such as most likely, optimistic and pessimistic adjusts estimates for bias by

the estimator and flaws in assumptions. You'll often see sales forecasts reported in this manner.

In training, adjustment is made for two reasons: first is conservatism. Conservatism is a guiding

principle of Phillips. It is also critical to state one is conservative in assumptions to build integrity

into your ROI model. Second, is for bias. Self-reported bias by participants is typically inflated. In

fact, studies done by organizations like the Tennessee Valley Authority (TVA) and separate

studies by KnowledgeAdvisors suggest that respondents tend to overestimate by a factor of 35%.

To this end, when computing an ROI calculation one might reduce the inputs by a factor of 35%

or a similar confidence rate as the adjustment factor for conservatism and bias.

Taken together, the principles of estimation, isolation and adjustment form a powerful model in

tabulating a systematic, replicable, and comparable ROI model for human capital.

The result of the process is a monetized benefit factor, that when multiplied by the salary (i.e. the

human capital) yields a monetized benefit from training. The model is easily adaptable, leveraging

automation and technology, to drill deep into a specific business result such as the ROI on sales,

quality, productivity, cycletime, customer satisfaction, or employee retention.

In order to measure ROI, it is necessary to build a picture of the effectiveness of the training

activities at various levels. The measurements will then provide the detailed data to calculate the

value of the training program, which in turn leads to the ROI calculation. A classic example of this

is an ROI model based on the Kirkpatrick model of training evaluation of 1959 and extended by

Phillips. “Evaluation” was defined as “measuring changes in behaviour that occur as a result of

training programs.”


Evaluation Levels (modified from the Kirkpatrick 4-level model)

Evaluation Level Description Characteristics

Level 1“Did they like it?” Measuring Reaction and

Identifying Planned Actions

Measures participants’ reaction to the program, and outlines specific

plans for implementation of learning to the job.

Level 2“Did they learn?” Measuring Cognitive Learning

and Retention

Measures skills, knowledge, or attitude changes as a result of the

training.

Level 3“Do they use it?”

Assessing Application of the program training on the job

Measures actual changes in behaviour on the job, and specific

applications of the training material.

Level 4“Did it impact the

bottom line?”

Identifying business results from the training

Measures the business impact of the training. (e.g. measures changes in

output, quality, costs, time, productivity or other business

metrics)

Level 5“What is the return

on learning investment?”

Calculating Return on Investment

Compares the monetary value of the results with the costs for the

program.

In most organisations, measuring training effectiveness at all Levels 1-4 is not feasible for all

participants and for all projects. In practice, the organisation must set targets for the scope of

evaluations at each level, and for the critical training courses or programmes which have most

importance to the success of the organisation. Typical targets for measurement activities at each

level are:

Level 1: 100% of participants / courses provide effectiveness data

Level 2: 50-70% of participants / courses provide effectiveness data



Level 5: 5-10% of actual ROI is measured, and extrapolated to the overall program.

From this, we can see that most organisations will initially only fully calculate ROI in detail on a

sample of the courses and participants. It is therefore essential to identify those courses and

participants who are:


Representative of the overall population

Taking part in important or high impact programs

Able to accurately assess the impact of training on their jobs

Amenable to providing the full depth of data required

Note: This model describes ROI as “level 5”; please bear in mind that it is not necessarily

appropriate to carry out detailed analysis at all levels up to this, in order to derive an ROI; it may

be enough for the client to know that an 80% pass mark on a technical test (Level 2) is “good

enough” to justify the investment, assuming some numerical value can be put on the passing of

the tests.

During the lifetime of the training programme and evaluation, the data can be collected at various

times

immediately after training, Level 1 feedback should be easily accessible

within a reasonably short time, it should be possible to measure the level 2 performance

change

it will take longer for level 3 data to be meaningful, as this typically requires some period

of practical experience, applying the learning to the job, and gaining some fluency with

the skills learned

Once level 3 data can be reasonably collected, level 4 data can be either derived from

the level 3 data immediately in some cases, but in other cases it will require a further

waiting period to assess the new level of business operations performance, especially

where the new levels of business performance can only be measured after the new skills

have been applied for some time (e.g. a helpdesk technician may be able to apply new

customer care skills after some practice (level 3); but it may take some further time

before the impact of these new skills on customer satisfaction can be gauged (level 4)).

Converting Data to Monetary Values

To calculate the return on investment, data collected at Level 4 is converted to monetary values

to compare to program costs. This requires a value to be placed on each unit of data connected

with the program. Ten approaches are available to convert data to monetary values where the

specific technique selected usually depends on the type of data and the situation:

Output data is converted to profit contribution or cost savings. With this approach,

output increases are converted to monetary value based on their unit of contribution to

profit or the unit of cost reduction. These values are standard values, readily available in

most organizations.

The cost of quality is calculated and quality improvements are directly converted to

cost savings. These values are standard values, available in many organizations.


For programs where employee time is saved, the participant wages and benefits are used for the value of time. Because a variety of programs focus on improving the time

required to complete projects, processes, or daily activities, the value of time becomes an

important and necessary issue.

Historical costs and current records are used when they are available for a specific

variable. In this case, organizational cost data are utilized to establish the specific value

of an improvement.

When available, internal and external experts may be used to estimate a value for an

improvement. In this situation, the credibility of the estimate hinges on the expertise and

reputation of the individual.

External databases are sometimes available to estimate the value or cost of data items.

Research, government, and industry databases can provide important information for

these values. The difficulty lies in finding a specific database related to the situation.

Participants estimate the value of the data item. For this approach to be effective,

participants must be capable of providing a value for the improvement.

Soft measures are linked, mathematically, to other measures that are easier to

measure and value. This approach is particularly helpful when establishing values for

measures that are very difficult to convert to monetary values, such as data often

considered intangible like customer satisfaction, employee satisfaction, grievances, and

employee complaints.

Supervisors and managers provide estimates when they are both willing and capable of

assigning values to the improvement. This approach is especially useful when

participants are not fully capable of providing this input or in situations where supervisors

need to confirm or adjust the participant’s estimate.

Education and Training staff estimates may be used to determine a value of an output

data item. In these cases it is essential for the estimates to be provided on an unbiased

basis.

This step in the ROI model is very important and is absolutely necessary to determine the

monetary benefits from education and training programs. The process is challenging, particularly

with soft data, but can be methodically accomplished using one or more of the above techniques.

Tabulating Program Costs

The next step in the process is tabulating the costs of the program. Tabulating the costs

involves monitoring or developing all of the related costs of the program targeted for the ROI

calculation. Among the cost components that should be included are:

the cost to design and develop the program, possibly prorated over the expected life of

the program;


the cost of all program materials provided to each participant;

the cost of the instructor / facilitator, including preparation times as well as delivery time;

the cost of the facilities for the program;

travel, lodging, and meal cost for the participants, if applicable;

salaries plus employee benefits of the participants for the time they attend the program;

and administrative and overhead costs of the education and training function allocated in

some convenient way to the training program

In addition, specific costs related to the needs assessment and evaluation should be included, if

appropriate. The conservative approach is to include all of these costs so that the total is fully

loaded.

Calculating the ROI

The return on investment is calculated using the program benefits and costs. The cost / benefit

ratio is the program benefits divided by cost. In formula form it is:

BCR =

The return on investment uses the net benefits divided by program costs. The net benefits are

the program benefits minus the costs. In formula form, the ROI becomes:

ROI (%) = x 100

This is the same basic formula used in evaluating other investments where the ROI is traditionally

reported as earnings divided by investment. The ROI from some programs is high. For example,

in sales, supervisory, leadership, and managerial training, the ROI can be quite large, frequently

over 100%, while the ROI value for technical and operator training may be lower.

Identifying Intangible Benefits

In addition to tangible, monetary benefits, most education and training programs will have

intangible non-monetary benefits. Data items identified that are not converted to monetary values,

are considered intangible benefits. While many of these items can be converted to monetary

values, for various reasons they often are not, such as the process used for conversion is too

subjective and the resulting values lose credibility in the process. These intangible benefits are

the sixth measure reported in the ROI impact study report and may include:


Program Benefits

Program Costs

Net Program Benefits

Program Costs

increased job satisfaction

increased organizational commitment

improved teamwork

improved customer service

reduced complaints and

reduced conflicts

For some programs, these intangible, non-monetary benefits are extremely valuable, often

carrying as much influence as the hard data items.

Operating Standards and Philosophy

To ensure consistency and replication of studies, operating standards must be developed and

applied as the process model is utilized to develop ROI studies. It is extremely important for the

results of a study to stand alone and not vary with the individual conducting the study. The

operating standards detail how each step and issue of the process will be handled. Figure 4

shows the ten guiding principles that form the basis for the operating standards.

The guiding principles not only serve as a way to consistently address each step, but also provide

a much needed conservative approach to the analysis. A conservative approach may lower the

actual ROI calculation, but it will also build credibility with the target audience.


Ten Guiding Principles

1. When a higher level evaluation is conducted, data must be collected at lower levels.

2. When an evaluation is planned for a higher level, the previous level of evaluation

does not have to be comprehensive.

3. When collecting and analyzing data, use only the most credible sources.

4. When analyzing data, choose the most conservative among alternatives.

5. At least one method must be used to isolate the effects of the solution.

6. If no improvement data are available for a population or from a specific source, it is

assumed that little or no improvement has occurred.

7. Estimates of improvements should be adjusted for the potential error of the estimate.

8. Extreme data items and unsupported claims should not be used in ROI calculations.

9. Only the first year of benefits (annual) should be used in the ROI analysis of short

term solution.

10. Costs of the solution should be fully loaded for ROI analysis.


Section 5: ROI Implementation Concerns and Milestones

5.1 Implementation of ROI – the steps

Although progress has been made in the implementation of ROI, significant barriers can inhibit

the implementation of the process. Some of these barriers are realistic; others are actually myths

based on false perceptions. The key implementation issues are briefly discussed in this section.

Discipline and Planning

A successful ROI implementation requires much planning and a disciplined approach to keep the

process on track. Implementation schedules, evaluation targets, data collection plans, ROI

analysis plans, measurement and evaluation policies, and follow-up schedules are required. Only

a carefully planned implementation will be successful.

Responsibilities

There are two areas of responsibility when implementing ROI. First, the entire training and

education staff is responsible for measurement and evaluation. Whether they are involved in

designing, developing, or delivering programs, these responsibilities typically include:

ensuring that the needs assessment includes specific business impact measures;

developing specific application objectives (Level 3) and business impact objectives (Level

4) for each program;

focusing the content of the program on performance improvement, ensuring that

exercises, case studies, and skill practices relate to the desired objectives;

keeping participants focused on application and impact objectives;

communicating rationale and reasons for evaluation;

assisting in follow-up activities to capture application and business impact data;

providing technical assistance for data collection, data analysis, and reporting;

designing instruments and plans for data collection and analysis; and

presenting evaluation data to a variety of groups.

The second area of responsibility is regarding details for those involved directly in measurement

and evaluation, either on a full time basis or as a primary duty. The responsibilities around this

group involve six key areas:

1. designing data collection instruments;

2. providing assistance for developing an evaluation strategy;

3. analyzing data, including specialized statistical analyses;

4. interpreting results and making specific recommendations;


5. developing an evaluation report or case study to communicate overall results; and

6. providing technical support in any phase of the ROI process.

Staff Skills Development

Many training staff members neither understand ROI nor have the basic skills necessary to

apply the process within their scope of responsibilities. The typical training and development

program does not focus on business results; it focuses more on learning outcomes.

Consequently, staff skills must be developed to utilize the results based approach. Ten skill

sets have been identified as necessary for the implementation of the ROI.

1. Planning for ROI calculations.

2. Collecting evaluation data.

3. Isolating the effects of training.

4. Converting data to monetary values.

5. Monitoring program costs.

6. Analyzing data including calculating the ROI.

7. Presenting evaluation data.

8. Implementing the ROI process.

9. Providing internal consulting on ROI.

10. Teaching others the ROI process.

Building the Process

Building a comprehensive measurement and evaluation process is best represented as a puzzle

where the pieces of the puzzle are developed and put in place over time. Figure 1 depicts this

puzzle and the pieces necessary to build a comprehensive measurement and evaluation process.

The first piece of the puzzle is the selection of an evaluation framework, which is a categorization

of data. The balanced scorecard process (Kaplan and Norton, 1993) or the four levels of

evaluation developed by Kirkpatrick (1974) offer the beginning points for such a framework. The

framework selected for the process presented here is a modification of Kirkpatrick’s four levels to

include a fifth level: return on investment.


Next, an ROI process model must be developed showing how data is collected, processed,

analyzed, and reported to various target audiences. This process model ensures that appropriate

techniques and procedures are consistently utilized to address almost any situation. Also, there

must be consistency as the process is implemented.

The third piece of the puzzle is the development of operating standards. These standards help

ensure the results of the study are stable and not influenced by the individual conducting the

study. Replication is critical for the credibility of an evaluation process. Operating standards and

guiding principles allow for replication, so that if more than one individual evaluates a specific

program, the results are the same.

Next, appropriate attention must be given to implementation issues as the ROI process becomes

a routine part of the education and training function. Several issues must be addressed involving

skills, communication, roles, responsibilities, plans, and strategies.

Finally, there must be successful case studies describing the implementation of the process

within the organization, the value a comprehensive measurement and evaluation process brings

to the organization, and the impact the specific program evaluated has on the organization.

While it is helpful to refer to case studies developed by other organizations, it is more useful and

convincing to have studies developed directly within the organization.

The remainder of this chapter focuses on the individual pieces of the evaluation puzzle

developing a comprehensive ROI process.

Evaluation Framework

The ROI process described in this section adds a fifth level to the four levels of evaluation

developed by Kirkpatrick (1974) to measure the success of training. The concept of different

levels of evaluation is both helpful and instructive to understanding how the return on investment

is calculated.


Level Measurement Focus

1 Reaction, Satisfaction

and Planned Action

Measures participant’s reaction to, and satisfaction with, the

program and captures planned actions.

2 Learning Measures changes in knowledge, skills, and attitudes.

3Application and

Implementation

Measures changes in on-the-job behavior and progress with

planned actions.

4 Business Impact Measures changes in business impact variables.

5 Return on InvestmentCompares program monetary benefits to the costs of the

program.

Level 1, Reaction, Satisfaction, and Planned Action, measures the level of satisfaction from

program participants along with their plans to apply what they have learned. Almost all

organizations evaluate at Level 1, usually with a generic, end-of-program questionnaire. While

this level of evaluation is important as a customer-satisfaction measure, a favorable reaction does

not ensure that participants have learned new skills or knowledge.

Level 2, Learning, focuses on what participants learned during the program using tests, skill

practices, role plays, simulations, group evaluations, and other assessment tools. A learning

check is helpful to ensure that participants have absorbed the program material and know how to

use it properly. However, a positive measure at this level is no guarantee that what is learned will

be applied on the job. The literature is laced with studies showing the failure of learning to be

transferred to the job (e.g., Broad, 1998).

At Level 3, Application and Implementation, a variety of follow-up methods are used to

determine if participants applied on the job what they learned. The frequency and use of skills

are important measures at Level 3. When training of technology applications is conducted, Level

3 measures the utilization of the technology. While Level 3 evaluations are important to gauge

the success of the application of a program, it still does not guarantee that there will be a positive

business impact in the organization.

The Level 4, Business Impact, measure focuses on the actual results achieved by program

participants as they successfully apply what they have learned. Typical Level 4 measures include

output, quality, costs, time, and customer satisfaction. Although the program may produce a

measurable business impact, there is still a concern that the program may cost too much.

Level 5, Return on Investment, the ultimate level of evaluation, compares the monetary benefits

from the program with the program costs. Although the ROI can be expressed in several ways, it


is usually presented as a percent or cost benefit ratio. The evaluation chain is not complete until

the Level 5 evaluation is conducted.

While almost all education and training organizations conduct evaluations to measure

satisfaction, very few actually conduct evaluations at the ROI level. Perhaps the best explanation

for this situation is that ROI evaluation is often characterized as a difficult and expensive process.

When business results and ROI are desired, it is also very important to evaluate the other levels.

A chain of impact should occur through the levels as the skills and knowledge learned (Level 2)

are applied on the job (Level 3) to produce business results (Level 4). If measurements are not

taken at each level, it is difficult to conclude that the results achieved were actually caused by the

program. Because of this, it is recommended that evaluation be conducted at all levels when a

Level 5 evaluation is planned. This practice is consistent with the practices of benchmarking

forum members of the American Society for Training and Development as well as best-practice

corporate universities as identified in a study conducted by the American Quality and Productivity

Center.

The ROI Process Model

The calculation of the return on investment in education and training begins with the model where

a potentially complicated process can be simplified with sequential steps. The ROI process

model provides a systematic approach to ROI calculations. A step-by-step approach helps keep

the process manageable so that users can address one issue at a time. The model also

emphasizes the fact that this is a logical, systematic process which flows from one step to

another. Applying the model provides consistency between ROI calculations. Each step of the

model is briefly described below.

Planning the Evaluation

One of the most important and cost-savings steps in the ROI process is planning the

evaluation. By considering key issues involved in the evaluation process, time, money, and

frustration can be significantly reduced. There are four specific elements of the evaluation

process that should be considered during the initial stages of the planning process.

1. Evaluation purposes are considered prior to developing the evaluation plan. The

purposes will often determine the scope of the evaluation, the types of instruments used, and

the type of data collected. For example, when an ROI calculation is planned, one of the

purposes is to compare the cost and benefits of the program. This purpose has implications

for the type of data collected (hard data), type of data collection method (performance

monitoring), the type of analysis (thorough), and the communication medium for results (formal

evaluation report). For most programs, multiple evaluation purposes are pursued.

2. A variety of instruments are used to collect data. The appropriate instruments are

determined in the early stages of developing the ROI. Questionnaires, interviews, and focus


groups are common instruments. When deciding which instruments to use, those most fitting

to the culture of the organization and most appropriate for the setting and evaluation

requirements should be considered.

3. Training programs are evaluated at five different levels as illustrated in Figure 2. Data

should be collected at Levels 1, 2, 3, and 4, if an ROI analysis is planned. This helps ensure

that the chain of impact occurs as participants learn the skills, apply them on-the-job, and

obtain business results.

4. A final element in planning evaluation is the timing of the data collection. Sometimes,

pre-program measurements are taken to compare with post-program measures. In some

cases, multiple measures are taken at different times throughout the process. In other

situations, pre-program measures are unavailable and specific follow-ups are taken after the

program. The important issue is to determine the timing for the follow-up evaluation. For most

education and training programs, a follow-up is usually conducted from 3 to 6 months after the

program.

These four elements - evaluation purposes, instruments, levels, and timing - are all

considerations in selecting the data collection methods and developing the data collection

plan. Once this preliminary information is gathered, the data collection plan and ROI analysis

plan are developed.

The data collection plan - After the above elements have been considered and determined,

the data collection plan is developed. The data collection plan outlines in detail the steps to be

taken to collect data for a comprehensive evaluation. The data collection plan usually includes

the following items:

Broad areas for objectives are developed for evaluation planning; more specific program

objectives are developed later.

Specific measures or data descriptions are indicated when they are necessary to explain

the measures linked to the objectives.

Specific data collection methodologies for each objective are listed.

Sources of data such as participants, team leaders, and company records are identified.

The time frame in which to collect the data is noted for each data collection method.

Responsibility for collecting data is assigned.

The ROI analysis plan

The ROI analysis plan is a continuation of the data collection plan. This planning document

captures information on several key issues necessary to develop the actual ROI calculation.

The key issues include:


Significant data items, usually Level 4, business impact measures, but in some cases

could include Level 3 data;

The method for isolating the effects of the training and education program.

The method for converting data to monetary values.

The cost categories, noting how certain costs should be prorated.

The anticipated intangible benefits.

The communication targets – those to receive the information.

Other issues or events that might influence program implementation.

These two planning documents are necessary to successfully implement and manage the

ROI process.

5.2 Concerns and challenges in implementing ROI

Concerns with ROI

Although much progress has been made, the ROI process is not without its share of problems

and drawbacks. The mere presence of the process creates a dilemma for many organizations.

When an organization embraces the concept and implements the process, the management team

is usually anxiously waiting for results, only to be disappointed when they are not quantifiable.

For an ROI process to be useful, it must balance many issues, including feasibility, simplicity,

credibility, and soundness. More specifically, three major audiences must be pleased with the

ROI process to accept and use it.

Practitioners - For years, education and training practitioners have assumed that ROI could not

be measured. When examining a typical process, they found long formulas, complicated

equations, and complex models that made the ROI process appear to be too confusing. With this

perceived complexity, these practitioners could visualize the tremendous efforts required for data

collection and analysis, and more importantly, the increased cost necessary to make the process

work. Because of these concerns, practitioners are seeking an ROI process that is simple and

easy to understand so that they can easily implement the steps and strategies. Also, they need a

process that will not take an excessive amount of time to implement. Finally, practitioners need a

process that is not too expensive. With competition for financial resources, they need a process

that will not command a significant portion of the budget. In summary, from the perspective of the

practitioner, the ROI process must be user-friendly, time-saving, and cost-efficient.

Senior Managers / Sponsors / Clients - Managers, who must approve education and training

budgets, request programs, or cope with the results of programs, have a strong interest in

developing the ROI. They want a process that provides quantifiable results, using a method

similar to the ROI formula applied to other types of investments. Senior managers have a never-

ending desire to have it all come down to ROI calculations, reflected as a percentage. They, as

do practitioners, want a process that is simple and easy to understand. The assumptions made


in the calculation and the methodology used in the process must reflect their frame of reference,

experience, and level of understanding. They do not want, or need, a string of formulas, charts,

and complicated models. Instead, they need a process that, when necessary, they can explain to

others. More importantly, they need a process with which they can identify; one that is sound and

realistic enough to earn their confidence.

Researchers - Finally, researchers will only support a process that measures up to their scrutiny

and close examination. Researchers usually insist that models, formulas, assumptions, and

theories be sound and based on commonly accepted practices. Also, they want a process that

produces accurate values and consistent outcomes. If estimates are necessary, researchers

want a process that provides the most accuracy within the constraints of the situation, recognizing

that adjustments need to be made when there is uncertainty in the process.

The challenge is to develop acceptable requirements for an ROI process that will satisfy

researchers and, at the same time, please practitioners and senior managers.

Criteria for an Effective ROI Process

To satisfy the needs of the three critical groups described above, the ROI process must meet

several requirements. Ten essential criteria for an effective ROI process are outlined below.

These criteria were developed with input from hundreds of education and training managers and

specialists:

1. The ROI process must be simple, devoid of complex formulas, lengthy equations, and

complicated methodologies. Most ROI attempts have failed with this requirement. In an

attempt to obtain statistical perfection, several ROI models and processes have become too

complex to understand and use. Consequently, they are not being implemented.

2. The ROI process must be economical, with the ability to be easily implemented. The

process should have the capability to become a routine part of education and training without

requiring significant additional resources. Sampling for ROI calculations and early planning

for ROI are often necessary to make progress without adding new staff.

3. The assumptions, methodology, and outcomes must be credible. Logical, methodical steps

are needed to earn the respect of practitioners, senior managers, and researchers. This

requires a very practical approach for the process.

4. From a research perspective, the ROI process must be theoretically sound and based on

generally accepted practices. Unfortunately, this requirement can lead to an extensive,

complicated process. Ideally, the process must strike a balance between maintaining a

practical and sensible approach and a sound and theoretical basis for the procedures. This is

perhaps one of the greatest challenges to those who have developed models for the ROI

process.


5. An ROI process must account for other factors that have influenced output variables. One

of the most often overlooked issues, isolating the influence of an education program, is

necessary to build credibility and accuracy within the process. The ROI process should

pinpoint the contribution of the program when compared to the other influences.

6. The ROI process must be appropriate with a variety of programs. Some models apply to

only a small number of programs, such as productivity training. Ideally, the process must be

applicable to all types of education and training and other programs such as career

development, organization development, and major change initiatives.

7. The ROI process must have the flexibility to be applied on a pre-program basis as well as a

post-program basis. In some situations, an estimate of the ROI is required before the actual

program is developed. Ideally, the process should be able to adjust to a range of potential

time frames for calculating the ROI.

8. The ROI process must be applicable with all types of data, including hard data, which is

typically represented as output, quality, costs, and time; and soft data, which includes job

satisfaction, customer satisfaction, absenteeism, turnover, grievances, and complaints.

9. The ROI process must include the costs of the program. The ultimate level of evaluation

compares the benefits with costs. Although the term ROI has been loosely used to express

any benefit of education or training, an acceptable ROI formula must include costs. Omitting

or understating costs will only destroy the credibility of the ROI values.

10.Finally, the ROI process must have a successful track record in a variety of applications. In

far too many situations, models are created but never successfully applied. An effective ROI

process should withstand the wear and tear of implementation and prove valuable to users.

Because these criteria are considered essential, an ROI process should meet the vast majority, if

not all criteria. The bad news is that most ROI processes do not meet these criteria. The good

news is that the ROI process presented below meets all of the criteria

There is resistance to measure the ROI in training, and the following are some of the questions

asked in the industry:

How can I measure ROI to determine the value added to the organisation?

What are the steps to measure ROI?

How often should ROI be measured?

In which areas of learning and change in behaviour can ROI be measured?

How do I implement the ROI process in the organisation?

How do I attach a rand value to learning?

What are the benefits of measuring ROI?


How do managers overcome resistance to measuring ROI?

Training managers know that ROI must be measured, yet very few are actually doing it. The

following are some of the reasons for not calculating ROI that have been provided by training

managers during recent conferences organised by ASTD Global Network South Africa:

There is a lack of a measurement culture in training departments. Managers do not know where

to start off they don't have the resources to calculate the ROI. It is perceived as too difficult,

complicated, too much effort or time consuming. They are so busy with all their training

programmes and SETA requirements, that there is no time to calculate ROI. They fear that if ROI

is calculated, it will show that their training adds no value and that they will be at risk to lose their

jobs!

Difficulties in calculating ROI

One of the common difficulties encountered in calculating ROI is determining the "shelf-life" of the

training program. How many times will a course will be used or "cycled" before business

strategies or budgets dictate its demise, technology renders it obsolete, or all eligible employees

have received the training? Changes fueled by technology, global competition, legal and

regulatory agencies, employee turnover and other factors make it increasingly difficult to

accurately predict the life cycle or shelf life of a training program. A conservative approach should

be used in evaluating the shelf life of a training program due to the rapid changes occurring in the

workplace.

It is difficult enough to get managers to send employees to training without imposing additional

requirements to collect pre- or post-training data to document the effects of training. HR

managers should make data collection as simple and effortless as possible and provide forms

and administrative assistance whenever possible. Collaboratively work with IT, HRIS, Production,

Engineering, Sales, Payroll, and Accounting personnel to "mine" existing data or develop ways to

collect, compile and extract the necessary data for a cost / benefit analysis of training.

The presence of unions could also create data collection problems. Frequently, unions are

concerned that the employees or union members are being evaluated. It is a good idea to get

union leaders to "buy in" to the idea of a ROI assessment. At the very least, a careful review of

the union contract is recommended along with clear communication regarding the goals and

scope of the training assessment.

Some behaviors or skills may be useful for only short periods of time while others, such as time

management, may continue for a lifetime and affect numerous workplace and personal activities.

Frequency of use of the newly acquired skills is also a factor in determining how long the skills


are retained. Therefore, determining or estimating the scope and duration of the benefits of

training can be a daunting task.

Measuring the impact of training for new products, technology, and services is difficult for two

reasons. First, if no prior training existed or no prior performance measures are available, it is

very difficult to ascertain the impact of training. Second, if prior measures of performance are

available, how does one determine the amount of improvement that is attributable to training as

opposed to how much is attributable to the new innovation? Frequently, this determination is a

judgment call or an intuitive decision rather than one based on factual data.

Fear and Misconceptions

Education and training departments often do not pursue ROI because of fear of failure or fear of

the unknown. A concern may exist about the consequence of a negative ROI. Staff members

may feel threatened. The ROI process also stirs up the traditional fear of change. This fear is

often based on unrealistic assumptions and a lack of knowledge of the process and is so strong

that it becomes a realistic barrier to many ROI implementations.

The false assumptions about the ROI process, will keep training and education staff members

from attempting ROI. Following are typical faulty assumptions.

Managers do not want to see the results of education and training expressed in monetary

values. In many cases, managers do want to see the results of education and training

expressed in monetary terms. But for years, they were told it could not be done. As

mentioned earlier, there is a greater awareness among executives that education and

training can be measured like other business practices. This awareness is increasing

interest and requests for these measures.

If the CEO does not ask for the ROI, he or she is not expecting it. This assumption

captures the most innocent bystanders. As CEOs become more aware of the process to

calculate the ROI in training, they want to begin seeing results immediately. The ROI

process is not a quick fix and takes time to learn and fully integrate into a training

organization. With this in mind, many organizations are developing ROI impact studies

long before senior management ask for it.

”As the manager of education and training, I have a professional, competent staff;

therefore, I do not have to justify the effectiveness of our programs”. This may be true for

some organizations – at least at the present. But with the ongoing changes in the

business community, it is becoming less and less common to see any function not

required to show its bottom-line impact. When budget allocations take place, the

functions with a “place at the table” are usually the ones that prove their contribution.

The training and development process is a complex, but necessary, activity;

consequently, it should not be subjected to an accountability process. The training and


development process is complex. And like other complex processes can be quite costly

to the organization. With this in mind, many organizations are holding the training and

education function to the same standard of accountability as other processes.

These false assumptions and others must be addressed and analyzed so they do not impede the

progress of ROI implementation.

5.3 Important milestones for ROI implementation

It is important to follow proper guidelines for the effective application of the ROI measurement

process by adhering to the following steps:

Create awareness for ROI in the organisation.

Build capacity for ROI by training staff to understand ROI.

Quantify information before the training in order to obtain a baseline (e.g. number of calls

answered, number of customer complaints, etc.).

Covert this data to monetary value (e.g. the cost of answering a call).

Allocate resources for ROI.

Develop a culture of measurement and accountability among training staff.

Start with only one course as a pilot programme to practise ROI skills.

Communicate results to training staff and the whole organisation.

Design improvement plans for training programmes in order to increase ROI.

Once ROI results are available, use the data to market future learning programmes.

HR managers can improve the actual ROI of training by utilizing the following tips:

Understand that business needs will change over the duration of the project. The longer it

takes to do a project, the more likely that change will occur. Be prepared to switch gears

or re-tool. Stay alert for indications of learning gaps or obsolescence of course materials

created by changes in technology, economic factors, or employee demographics.

Use the experience of senior managers and employees in estimating the results of

training. Estimating requires a little math and lots of experience.

Anticipate and plan for glitches. They will happen. Do not get discouraged. Deal

realistically with unexpected events and surprises. Some of them will be problems that

may reduce expected outcomes. Others may stimulate creativity and produce very

positive results.


Keep an eye on costs and monitor any deviations from estimated or budgeted

expenditures. Quick action or adjustments may enable the HR manager to curtail or

reduce costs and keep the project within or near budget.

Balance short term and long-term goals. The pressure to deliver consistent short-term

financial performance may limit the resources available for investment in growth

opportunities, such as the "development" of employees for future roles.

Secure managers' and supervisors' support for training. Effective training depends on

three persons: the trainer, the trainee and the supervisor or manager. All three must

agree on expected outcomes and how and when they will be measured.

Align training initiatives with strategic business plans. Training should improve your

organization's competitive edge, which may be efficiency, first class service, or creativity

and innovation.

Track training costs by employee for career pathing and succession planning

HR must shift its emphasis from being the "keeper and distributor of people information" to the

more critical role of "developer of people and productivity." Training benefits must be greater than

the cost of the training. Training must ultimately improve the profitability of an organization. The

four-step process for calculating ROI enables HR mangers to determine the productivity affect of

training or its impact on the bottom line. Reducing training to save money may be like stopping a

clock to save time.


Section 6: ROI Lessons from Practice

6.1 Case studies related to ROI

This section contains two case studies related to ROI in training and can be used to help develop

your understanding and application of the tools and processes involved in this new field.

Case study 1: Reliance Financial Services

At the end of a monthly staff meeting, Frank Thomas, CEO of Reliance Financial Services,

asked Margaret Bongani, Manager of Training and Development, about the Effective

Interpersonal Skills Workshops that had been conducted with all supervisors and managers

throughout the company. The workshop featured the Myers-Briggs Type Indicator (MBTI) and

showed participants how to interact with, and understand, each other in their routine activities.

The MBTI classifies people into one of 16 personality types.

Frank continued, “I found the workshop very interesting and intriguing. I can certainly identify

with my particular personality type, but I am curious what specific value these workshops have

brought to the company. Do you have any way of showing the results of all 6 workshops?”

Margaret quickly replied, “We certainly have improved teamwork and communications

throughout the company. I hear people make comments about how useful the process has been

to them personally.” Frank added, “Do we have anything more precise? Also, do you know how

much we spent on these workshops?” Margaret quickly responded by saying, “I am not sure

that we have any precise data and I am not sure exactly how much money we spent, but I can

certainly find out.” Frank concluded with some encouragement, “Any specifics would be helpful.

Please understand that I am not opposing this training effort. However, when we initiate these

types of programs, we need to make sure that they are adding value to the company’s bottom

line. Let me know your thoughts on this issue in about two weeks.”

Margaret was a little concerned about this CEO’s comments, particularly since the CEO enjoyed

the workshop and has made several positive comments about it. Why was he questioning the

effectiveness of it? Why was he concerned about costs?

These questions began to frustrate Margaret as she reflected over the year and a half period in

which almost every senior member of staff had attended the workshop. She recalled how she

was first introduced to the MBTI. She attended a workshop conducted by a friend, was

impressed with the instrument, and found it to be helpful as she learned more about her own

personality type.


Margaret thought the process would be useful to Reliance managers and asked the consultant

to conduct a session internally with a group of middle and senior staff members. With

favourable reaction, she decided to try a second group with the top team, including Frank

Thomas. Their reaction was favourable. Then she launched it with the entire staff, using

positive comments from the CEO, and the feedback has been excellent.

She realized that the workshops have been expensive and over 60 staff had attended.

However, she thought that teamwork has improved, but there is no way of knowing for sure.

With some types of training you never know if it works, she thought. Margaret believed that she

had done her homework on this one, but was still facing a dilemma. Should she respond to the

CEO or just ignore the issue?

Reliance Financial Services ~ Part B

After some intense discussions with Frank Thomas, the CEO, around a structured approach to

training, Margaret conducted a detailed needs assessment in the organisation. The assessment

uncovered four key problems that could be corrected with improved supervisor and managerial

leadership skills. These included excessive employee turnover, excessive absenteeism,

deteriorating quality of proposals processed, and sagging productivity per professional clerk

employed. Each of those four measures are monitored in the work unit under the control of a

supervisor or manager. Quality and productivity are monitored daily and reported weekly.

Absenteeism is reported weekly, and turnover is reported monthly, using annualised figures.

As a result of the needs assessment, a three-day leadership workshop for all supervisors and

managers was developed and implemented. The workshop focuses on building specific

leadership skills to enable supervisors and managers to correct the problem areas identified in

the needs assessment and secure significant improvements in all four key output variables.

The specific objectives of the leadership program are as follows. After attending this program,

participants should:

Be able to identify and discuss various leadership models and theories.

Be able to select the appropriate leadership style for a particular situation.

Serve as a positive role model for appropriate leadership behaviour.

Apply specific leadership skills.

Reduce employee turnover from an average annual rate of 12% to an industry average of

7.3% in one year.

Reduce absenteeism from a weekly average of 3% to 1% in six months.

Improve quality by 15% in six months.

Increase productivity by 12% in six months.


Case study 2:

ROI Profile: IBM Mindspan Solutions: IBM Basic Blue

Bottom Line

IBM's Basic Blue e-learning initiative brings training to more than 5000 IBM managers annually.

Using a blended approach of e-learning, electronic community, coaching, and simulation, IBM

was able to not only reduce costs by moving training to the Web, but take advantage of the

flexibility of the electronic medium to provide a richer learning environment for students.

The company

IBM's name is synonymous with computing. From its predecessor the Computer-Tabulating-

Recording company's establishment in 1911 to its launch of the personal computer in 1981,

IBM's goal has been to lead in the creation, development, and manufacture of advanced

information technologies. IBM sales of computer systems, software, networking systems,

storage devices, microelectronics, and services earned revenues of nearly US$90 billion in 2000.

In the turbulent IT market where vision and flexibility are key, IBM has recognized the importance

of training its managers both on IBM's strategy and culture and on leadership and people

management.

The challenge

As part of its ongoing management development program, IBM trains more than 5000 new

managers each year. Traditionally, managers were brought together for a 5-day event to

learn the basics on IBM culture, strategy, and management. As the complexity of their jobs

increased, IBM recognized that five days was not enough time to train managers effectively

- and that to help managers evolve with the industry, training needed to be a ongoing

process instead of a one-time event.

In 1996 Nancy Lewis, Director of IBM Management Development, began to develop IBM's

global management training program. Recognizing that more than one session was needed

- but that bringing together 5000 people from around the world was costly and time-

consuming, IBM looked at e-learning technology, and specifically, IBM's own Mindspan

solution, to support its manager training. The goal was to find appropriate technology to

support different parts of the manager training process - and engage and teach people who

were used to face-to-face training. "We like saving money and a quick payback but that's

not why we did this," said Nancy, "we did it for effectiveness of our managers around the

globe."


The strategy

IBM developed a 4-tier e-business learning approach, Basic Blue for Managers, which blends

technologies to support regular classroom sessions. The management training program

includes a Web-based learning infrastructure, virtual collaborative tools, content references,

and interactive online simulators to complement face-to-face instruction. Managers

participate in 26 weeks of self-paced on-line learning delivered through Lotus LearningSpace

modules. At the end of the self-paced learning, managers meet for their 5-day face- to-face

session to address higher-level issues and skills -and continue with on-line collaboration

with other managers as new needs or issues arise

IBM's 4-tier model focused on information and awareness through "QuickView" reference

guides, understanding and practice through interactive modules and simulation, learning

from experts through same-time awareness and col laboration, and higher order skills

through virtual learning labs and face -to-face meetings. IBM's Basic Blue for Managers is now

a 12-month training process with 5 times as much content as the previous new manager

training program. The program has grown to include approximately 6400 managers

Key benefit areas

Basic Blue for Managers, now an IBM Mindspan solution, enables IBM to train managers to

lead high-performance teams - without the expense of on-site meetings and travel. Key

benefits Nucleus measured in calculating the ROI from the solution include the following:

• Direct savings such as reduced program, travel, and manager off-site costs. Because

75 percent of the training is delivered through distance learning and 25 percent is in a

traditional classroom, IBM has been able to significantly reduce the costs associated with

training while increasing the amount of content taught.

• Reduced the direct cost of content development. In addition to the savings with the Basic

Blue project, Reusable templates have enabled other IBM divisions to develop content

without re-development of the training platform.

Indirect savings in the form of increased manager productivity. The self-service nature of

Basic Blue allows managers to better utilize their time. IBM also estimates managers are

able to reduce the time needed to learn new material by 25%.

IBM also tracks a number of additional benefits from Basic Blue. Measurement of the impact of

the enhanced training on managers' performance has shown that IBM's blended approach to

initial and ongoing training has enabled its managers to make sustained behavior changes

which then lead to significant business performance improvements


Key cost areas

Infrastructure costs for IBM Basic Blue included the cost to purchase software, hardware,

hosting services, and IT support. As with any e-learning initiative the cost of the infrastructure is

only a part of the total cost. IBM invested additional resources to create the training and

simulation modules that would match the needs of the company. In doing so, IBM ensured a

smooth transition from the traditional learning approach to the new blended e-learning approach.

Nucleus believes the company's success with its Basic Blue initiative is due in large part to the

commitment the company made early in the project to invest proper resources to develop high

quality learning modules. The success of the Basic Blue application shows the value of the

combination of both technology and the proper level of training services.

Lessons learned

By combining a variety of mediums with traditional classroom teaching, IBM found it was able to

reduce training costs - and that managers preferred and retained more from the "blended"

approach than from the classroom alone. IBM learned that longstanding guidelines for successful

innovations in other industries held true for e-learning: (1) simplicity — in navigation and usability

— enhanced the student experience; (2) compatibility with existing technologies sped student

acceptance; and (3) trialability — "safe" opportunities to try online learning, and positive, engaging

experiences early on -- increased adoption speed. Anytime / anywhere access to

performance-support was perceived by students as the key advantage of IBM's blended

learning.

IBM also found that to be effective, the technology had to closely match the needs of the users.

Even though its manager group was technology literate, rather than forcing them

To adapt to the technology, IBM took steps to model the application to match the managers'

learning styles. This continues to be an ongoing process as IBM adjusts the learning

environment based on feedback from users and the evolution of technology.

Bob MacGregor, Manager, IBM Management Development, advises that when venturing into e-

learning for the first time, teams can get bogged down in studying what to do. He says that,

"equipped with the vision of what would work best for students, the Basic Blue team jumped in

and did it."

Calculating the ROI

IBM's goals were to improve the training process for better-performing managers as well as

manage training. Reduction in costs that would have been required to increase training levels

without distance learning made up the lion's share of the return from IBM's investment.


When estimating the ratio of direct and indirect benefits, Nucleus assumed all savings in

employee time or increased productivity were indirect. Direct savings included reduced travel and

teacher expenses along with other direct savings

6.2 Lessons from practice related to ROI

Training ROI & effectiveness: How can I measure training effectiveness or its return on

investment? One of the keys is to have a defined plan / process when you go or send a staff


member to training; a plan for before training, during training and what to do after training.

Training should be managed so it can be planned & linked to the business results. Goals must be

clear. Mechanisms should be in place to provide re-enforcement to the person who is learning for

his / her efforts to implement what they have learned.

Using the PDSA process is one way. For example:

PLAN How does this training relate to the aim of the organization?

Aim of the division or business unit?

To the person's developmental plan?

How does the training relate to the customer?

Where does it fit into my long term growth?

Why is it important?

Determine the expectations during training and more important after training.

What are the expected outcomes?

How will I practice or demonstrate my new skills and knowledge.

Create understanding of the linkage between the training and the day to day activities.

DO Attend training.

Monitor progress throughout the training.

Participate

Develop an ideas list

Take notes

Network with class mates

Develop a plan for application of ideas.

Teach or share your new skills with someone else.

Maintain contact with network buddies.

STUDY What was learned?

How will the learning that took place be applied?

Review the linkage to the job and the aim of the organization and the customer

Analyze / track to see to what extent learning actually created the intended business

results.

ACT


Integrate the learning into the day to day activities in order to achieve positive business

results.

Continue to monitor or study the results of training over time (critical data for justification

for future training investment)

For example: Improved service levels; increased morale and job satisfaction; improved

performance / production; improve efficiency (greater production / less staff)

Continue the PDSA thought process…..

Remember, when compiling an approach for ROI in learning; keep these best practices

concluding thoughts in the top of your mind:

Best Practice #1: Plan your metrics before writing survey questions.First and foremost, never ask a question on a data collection instrument unless it ties to a metric

you will utilize. As simple as this sounds, often is the case where organizations create questions

with no purpose in mind.

Best Practice #2: Ensure the measurement process is replicable and scalable. Organizations tend to spend thousands of dollars on one-off projects to measure a training

program in detail. This information is collected over many months with exhaustive use of

consultants and internal resources. Although the data is powerful and compelling, management

often comes back with a response such as 'great work, now do the same thing for all the training.'

Unfortunately such one-off measurement projects are rarely replicable on a large-scale basis. So

don't box yourself into that corner.

Best Practice #3: Ensure measurements are internally and externally comparable.Related to best practice #2 is the concept of comparability. It is a significantly less powerful

endeavor to do a one-off exercise when you have no base line of comparability. If you spend

several months calculating out a 300% ROI on your latest program how do you know if that is

good or bad? Surely a 300% ROI is a positive return but what if the average ROI on training

programs is 1000%?

Best Practice #4: Use industry-accepted measurement approaches.Management is looking to the training group to lead the way in training measurement. It is the job

of the training group to convince management that their approach to measurement is reasonable.

This is not unlike a finance department that must convince management of the way it values

assets. In both cases, the group must ensure the approach is based on industry accepted

principles that have proof of concept externally and merit internally.


Best Practice #5: Define value in the eyes of your stakeholders.If you ask people what they mean by 'return on investment' you are likely to get more than one

answer. In fact, odds are you'll get several. Return on investment is in the eyes of the beholder.

To some it could mean a quantitative number and to others it could be a warm and fuzzy feeling.

Best Practice #6: Manage the change associated with measurement.As you can likely see from some of the best practices, they might be doomed for failure if you fail

to manage the change with your stakeholders. Successful organizations will spend considerable

time and energy planning for the change. Assess the culture and the readiness for change. Plan

for change or plan to fail.

Best Practice #7: Ensure the metrics are well balanced.Although you want to understand the needs of your stakeholders and have them define how they

perceive value, you also need to be proactive in ensuring that your final 'measurement scorecard'

is well balanced.

Best Practice #8: Leverage automation and technology.Although this goes hand and hand with a measurement process that is replicable and scalable it

is worthy of separate mention. Your measurement process must leverage technology and

automation to do the heavy lifting in areas such as data collection, data storage, data processing

and data reporting.

Best Practice #9: Crawl, walk, run.When designing a learning measurement strategy it is nice to have a long term vision, but don't

attempt to put your entire vision in place right out of the blocks. The best approach is to start with

the low hanging fruit that can be done in a reasonable time frame to prove the concept,

demonstrate a 'win' and build a jumping off point to advance it to the next level.

Best Practice #10: Ensure your metrics have flexibility.The last thing you want to do is rollout a measurement process that is inflexible. You will likely

have people who want to view the same data but in many different ways. You need to have

architected your database to accommodate this important issue thereby creating measurement

flexibility.


References

Barney, M, Measuring ROI in Corporate Universities, Death of Student Day and Birth of Human

Capital

Broad, M.L., (1999). Transferring Learning to the Workplace. Alexandria, VA: American Society

for Training and Development.

Cascio, Wayne F., Costing Human Resources: The Financial Impact of Behavior in

Organizations, South-Western Publishing, 1999

Donald L. Kirkpatrick, Evaluating Training Programs: The Four Levels, 2nd Edition, Berrett-

Koehler Publishers, Inc, San Francisco

Fitz-enz, Jac, ROI of Human Capital, AMACOM, American Management Association, New York,

2000

Geber, B. (1994, February). A Clean Break for Education at IBM, Training, 33-36.

Industry Report 1999. (1999, October). Training, 34(10), 33-75.

Jack J. Phillips, Return on Investment in Training and Performance Improvement Programs, Gulf

Publishing, Houston, Texas

Kirkpatrick, Donald, Evaluating Programs: The Four Levels, San Francisco: Berrett-Koehler 1998

Kirzan, William G., "Labor: Craft Training Pays Off Quickly," Engineering News Record, New

York, September 28, 1998.

Kimmerling, G. (1993, September). Gathering Best Practices. Training and Development, 47,(3),

28-36.

Kirkpatrick, D. L., Techniques for Evaluating Training Programs, Evaluating Training Programs

(pp. 1-17) Alexandria, VA: ASTD,

Phillips, Jack J., Return on Investment in Training and Performance Improvement, Gulf

Publishing, Houston Texas, 1997

Phillips, J. J. (1994). In Action: Measuring Return On Investment, Vol. 1. Alexandria, VA:

American Society for Training and Development.

Phillips, J. J. (1996a, March). Was it the training? Training & Development, 50(3), 28-32

Phillips, J. J. (1996b, April). How much is the training worth? Training & Development, 50(4),

20-24.


Phillips, J. J. (1997a). Return On Investment In Training and Performance Improvement

Programs. Houston, TX: Gulf Publishing.

Phillips, J. J. (1997b). In Action: Measuring Return on Investment, Vol. 2. Alexandria, VA:


Phillips, J. J. and Pulliam, P. F. (1999, May / June). Dispelling the ROI Myths. Corporate

University Review (pp. 32-36).

Phillips, J. J. and P. P. (2000c). In Action: Measuring Return on Investment, Vol. 3. Alexandria,

VA: American Society for Training Development.

Phillips, J. J. (1998). Implementing Evaluation Systems and Processes. Alexandria, VA.


Phillips, J. J. (2000a). The Consultant’s Scorecard. New York, NY. McGraw-Hill.

Phillips, J. J., Ron D. Stone, Patricia Pulliam Phillips (2000b). the Human Resources Scorecard:

Measuring the Return on Investment. Houston, TX. Gulf Publishing

Phillips, Patricia Pulliam and Holly Burkett (2000). ROI on a Shoestring. ASTD Infoline Series.

Alexandria, VA: American Society for Training and Development.

Sibbet, David. (1997) 75 Years of Management Ideas and Practice, 1922-1997. Harvard

Business Review. Supplement.

(2000). The Corporate University: Measuring the Impact of Learning. Houston, TX: American

Quality and Productivity Center.

William and Mary Business Review. (1995). Corporate Training: Does it pay off? Supplement.


Recommended Reading

The Consultant’s Scorecard: Tracking Results and Bottom-line Impact of Consulting

Projects, Jack J. Phillips, McGraw-Hill, New York, New York, 2000

The Human Resources Scorecard: Measuring the Return on Investment in Human

Resources, Jack J. Phillips, Ron D. Stone, and Patricia P. Phillips, Gulf Publishing, Houston,

Texas, 2000.

ROI on a Shoestring, Phillips, Patricia P. and Burkett, Holly

InfoLine Series on Evaluation, Jack J. Phillips, (Editor), American Society for Training and

Development, Alexandria, Virginia, 1999.

Volume 1 – Level 1 Evaluation: Reaction and Planned Action Issue # 9813

Volume 2 – Level 2 Evaluation: Learning Issue # 9814

Volume 3 – Level 3 Evaluation: Application Issue # 9815

Volume 4 – Level 4 Evaluation: Business Results Issue # 9816

Volume 5 – Level 5 Evaluation: ROI Issue # 9805

In Action: Implementing Evaluation Systems and Processes, Jack J. Phillips, (Editor),

American Society for Training and Development, Alexandria, Virginia, 1998

Evaluating Training Programs (2nd Edition), Donald L. Kirkpatrick, Berrett-Koehler

Publishers, San Francisco, California, 1998.

Return on Investment in Training and Performance Improvement Programs, Jack J.

Phillips, Gulf Publishing, Houston, Texas, 1997.

Evaluating the Impact of Training, Scott B. Parry, American Society for Training and

Development, Alexandria, Virginia, 1997

Hand book of Training and Evaluation and Measurement Methods (3rd Edition), Jack J.

Phillips, Gulf Publishing, Houston, Texas, 1997.

In Action: Measuring Return on Investment, Volume 2, Jack J. Phillips, (Editor),

American Society for Training and Development, Alexandria, Virginia, 1997.

Accountability in Human Resource Management, Jack J. Phillips, Gulf Publishing,

Houston, Texas, 1996.

Evaluating Training Effectiveness, Second Edition, Peter Bramley, McGraw-Hill Book

Company, London, 1996.


Evaluating Human Resources, Programs, and Organizations, Byron R. Burnham,

Krieger Publishing Company, Malabar, Florida, 1995.

In Action: Measuring Return on Investment, Volume 1, Jack J. Phillips, (Editor),

American Society for Training and Development, Alexandria, Virginia, 1994.

Return On Quality, Roland T. Rust, Anthony J. Zahorik, Timothy L. Keiningham, Irwin

Publishers, Chicago, Illinois, 1994

Make Training Worth Every Penny, Jane Holcomb, Pfeiffer & Company, San Diego,

California, 1994.

The Training Evaluation Process, David J. Basarab, Sr. and Darrell K. Root, Kluwer

Academic Publishers, Norwell, Massachusetts, 1992.

Training Evaluation Handbook, A. C. Newby, Pfeiffer & Company, San Diego, California

1992.

Handy Resources

http://www.fasset.org.za

http://www.businessballs.com/

http://www.ibmweblectureservices.ihost.com/servlet/Gate/Component?

action=load&customer=ibm&offering=sngl&itemCode=ltu2153f

http://www.learning-designs.com/page_images/LDOArticleBottomLineonROI.pdf

http://www.learningcircuits.org/2001/jan2001/cross.htm

http://hub.col.org/2002/collit/att-0073/01-The_CIPP_approach.doc

Useful links

http://www.franklincovey.com/jackphillips/index.html

http://www.franklincovey.com/jackphillips/satest.htm

http://www.glaworld.com/documents/2F1D3F14-1968-4539-9DC1213B0B4DB724/

Human%20Capital%20ROI.pdf


http://www.glaworld.com/documents/2F1D3F14-1968-4539-9DC1213B0B4DB724/Human%20Capital%20ROI.pdf

http://www.glaworld.com/documents/2F1D3F14-1968-4539-9DC1213B0B4DB724/Human%20Capital%20ROI.pdf

http://www.franklincovey.com/jackphillips/index.html

http://hub.col.org/2002/collit/att-0073/01-The_CIPP_approach.doc

http://www.learningcircuits.org/2001/jan2001/cross.htm

http://www.learning-designs.com/page_images/LDOArticleBottomLineonROI.pdf

http://www.ibmweblectureservices.ihost.com/servlet/Gate/Component?action=load&customer=ibm&offering=sngl&itemCode=ltu2153f

http://www.ibmweblectureservices.ihost.com/servlet/Gate/Component?action=load&customer=ibm&offering=sngl&itemCode=ltu2153f

http://www.businessballs.com/

http://www.fasset.org.za/

fasset skills development · web viewequipment and facilities - including all types of av...

Documents