Measurement and Evaluation in Corporate Universities

By Mark Allen, Ph.D. and Philip McGee, Ed.D.

Although corporate universities in America can be

traced back to Pre-World War II times, the phenomenon truly

exploded in the 1990s. Among the reasons cited for the

proliferation of corporate universities was dissatisfaction

with the education that was provided in both degree and

non-degree programs by traditional universities. Yet many

corporate universities have adopted some of the trappings

of traditional universities: deans, course catalogues,

numerous schools within a university, and even a few

sprawling campuses. Yet one phrase from traditional academe

has not worked its way into corporate university jargon:

institutional research.

Many corporate universities do, however, invest

considerable resources in measurement and evaluation.

Corporate training managers have a long history of

measuring the effectiveness of training programs, and

corporate university deans and directors have devised

numerous clever and creative means for evaluating the

overall value of a corporate university.

In this chapter, we will first define corporate

universities and discuss how they differ from both

traditional universities and traditional training

departments. The rest of the chapter is devoted to a

discussion of a variety of methods that are used to measure

both individual programs as well as entire corporate

universities.

Definition of Corporate Universities

Just as many state colleges underwent name changes in

the past two decades and became universities, corporate

colleges, as they were called in the 1980s (Eurich, 1985)

transformed into corporate universities by the 1990s.

Entities that called themselves corporate universities

ranged from very large organizations (such as Motorola

University) to small training departments that sought

greater status through a name change. For many years, there

was little agreement about what constituted a corporate

university. Many companies (such as General Electric)

avoided using the term, and in Europe, where the phenomenon

is growing rapidly, the term is almost never used

corporate academies is the preferred appellation, and the

word university in many countries is exclusively reserved

for degree-granting institutions (Renaud-Coulon, 2002). In

this chapter, we define a corporate university as,

an educational entity that is a strategic tool

designed to assist its parent organization in

achieving its mission by conducting activities that

cultivate individual and organizational learning,

knowledge, and wisdom (Allen, 2002, p. 9).

While the definition stipulates that corporate

universities are educational in nature, the most important

word in this definition is strategic. A corporate

university, in order to fit this definition, must be truly

strategic in its intent and activities. As the definition

implies, the corporate university exists to help its parent

organization (and a corporate university, by definition, is

not a stand-alone entityit is a part of a larger

organization) achieve its mission. The activities of a

corporate university are related to the overall strategy

and mission of the organization it serves.

The definition does not specify what sort of

activities a corporate university undertakes. The most

frequently asked question we receive is how does a

corporate university differ from a training department?

The best answer is that a training department does

training, while a corporate university does training, plus

many other things. Among the activities that corporate

universities are responsible for or assist with are:

needs assessment

managerial and executive development

the design, delivery, and management of e-

learning programs

internal and possibly external marketing of

programs

university partnerships

strategic hiring

new employee orientation

career planning

succession planning

knowledge management

research and development

culture change

strategic change, and, of course

measurement and evaluation.

This list goes well beyond the scope of most training

departments. While training is generally highly tactical,

this list of corporate university activities is much more

strategic.

The final piece of the definition discusses

individual and organizational learning, knowledge, and

wisdom. The first key distinction is between the

individual and the organization. While learning activities

at traditional universities are generally focused on

individuals, learning in corporate universities is designed

to benefit both the individuals in the classroom and the

organization itself. As the concept of the learning

organization (Senge, 1990) grew in popularity throughout

the 1990s, organizations began focusing their efforts on

organizational learning. Of course an organization cannot

learn unless the people that comprise the organization are

learning, so corporate universities must focus on both

individual and organizational learning. This is a crucial

distinction between corporate universities and traditional

universities, where the main locus of educational efforts

is individual students. While much of the classroom and e-

learning activity in corporate universities is directed at

individuals, it is important to remember that the raison

dtre of corporate universities is organizational

betterment.

Learning is a process of changing, both for

individuals and organizations. Usually, that change process

involves the acquisition of knowledge, which may be

declarative (facts, or things you know) or procedural

(skills, or things you know how to do). The final piece of

the definition, wisdom, refers to the intelligent

application of knowledge. The organization does not benefit

if knowledge is acquired, but not used. We will elaborate

on this concept in our discussion of behavior change and

organizational impact below, but without the wisdom to

apply knowledge for positive results, the organization does

not receive any benefits from learning for its own sake.

This distinction marks another area of difference

between corporate and traditional universities. Academe has

long valued learning for its own sake, but that is not part

of the strategic nature of corporate universities.

This leads to another important distinction between

corporate and traditional universities: degree programs.

Nowhere in our definition or list of corporate university

activities does the granting of degrees appear. There are

thousands of accredited degree granting universities in the

United Statescorporate universities do not need to get

into the business of establishing degree programs. As

corporate universities proliferated into the hundreds and

perhaps thousands over the past decade, the great fear in

the halls of traditional academe was that corporate

universities would encroach on their turf. It turns out

that corporate universities had a very ambitious to-do

list, but the granting of degrees was not on that list. In

2000, Gordon Thompson went in search of corporate

universities that were granting degrees. In a study of

corporate universities in the United States and Canada, he

found fewer than ten corporate universities that were

granting degrees in a population that was estimated to

exceed 2000 entities that called themselves corporate

universities (Thompson, 2000).

The other notable omission from the definition of

corporate universities is the term corporation. While

most corporate universities are in the for-profit world,

there are numerous corporate universities that are not

corporate at all--they exist in not-for-profit

corporations, municipalities, and governmental agencies.

Traditional Training Measures: Kirkpatricks Four Levels

Since corporate universities are direct descendants

from training departments, it is not surprising that the

basic model that many corporate universities use for

measurement is the Kirkpatrick Four Level Model

(Kirkpatrick, 1994) that has been favored by training

departments for more than four decades. Developed by Donald

Kirkpatrick as his doctoral dissertation in the late 1950s,

this model still serves as the basic template for

measurement in many training departments and corporate

universities.

In Kirkpatricks model, the first level of measurement

is reaction. Immediately after a training program,

participants are asked for their reactions to the classdid

you like the teacher? Did you find the content useful? Were

the meals tasty and the chairs comfortable? While this type

of measurement does not yield too much in the way of in-

depth data, it does have the advantage of being inexpensive

and easy to administer.

Of course the whole point of education is to create

learning, and Level One evaluation does not measure

learning. Kirkpatricks second level addresses learning.

Learning can be measured in a variety of ways: short answer

tests, essays, demonstrations, and even on the job.

A major difference between traditional and corporate

universities is in the area of measuring the

accomplishments of students. While most traditional

universities will measure using Kirkpatricks first two

levels, that is where they stop. It is customary after a

college course for students to fill out surveys about the

course and the professorin Kirkpatricks world, this is

Level I evaluation. College courses also attempt to measure

student learning, both throughout the course and upon its

completion. But most university evaluation ends when

individual courses end. Kirkpatricks final two levels

extend over periods of time after the completion of the

course. Measurement that occurs months or years after a

course ends is generally beyond the scope of traditional

universities, but is at the core of meaningful evaluation

in corporate settings.

The third level in Kirkpatricks taxonomy is behavior

change. It is not enough, in corporate learning

initiatives, to merely measure learning. Employees need to

do something differently in order for that learning to have

relevance for the organization. Learning without behavior

change is merely learning for its own sake, which is

generally not the province of corporate universities. In

order to measure behavior change, employees behaviors need

to be tracked prior to the training so a baseline is

established. Then, post-training, behavior needs to be

observed in order to establish the extent of change.

Measurement of behavior change, especially across a large

number of students, is not a simple task and requires a

high level of commitment by the organization and an

investment of time and energy.

Kirkpatrickss fourth level is results or

organizational impact. Not only do you want learning and

behavior change to happen, you also want those changes to

yield positive results for the organization. While the

locus of measurement for learning and behavior change is

typically the individual employee, results are typically

measured at the organizational level. There is often a

lengthy lag between the time of the training and the

measurement of results. Moreover, the organization needs to

determine what sort of results it is looking for prior to

the training (always a good idea), and track correlation

over time. Needless to say, causality is tricky here, but

most organizations that aspire to Level IV measurement (and

not all do), are satisfied with a correlation between

training and positive organizational results.

ROI and Human Capital Measures

While Kirkpatrick ended his model at Level IV, Jack

Phillips has done considerable work in the area of

measuring Return on Investment (ROI) for training programs

(Phillips, 1997). ROI has sometimes been called Level Five

evaluation. Like standard ROI measures, Phillips work

involves examining the ratio of costs to benefits. However

in the world of corporate universities, costs are not as

easily measured as most areas, and the quantification of

benefits can be extremely tricky.

Direct costs for a training class consist of payment

to the instructor, the cost of the room, course materials

and handouts, and catering. However the company also incurs

a cost for having a number of employees out of the

workplace. Most companies figure this cost as the salary

plus taxes and benefits for each worker in the room for as

many days as the class lasts.

Calculating the benefits is even more problematical.

First, the company needs a pretty clear picture of what it

is trying to accomplish. Then, it needs to figure out what

a successful outcome would look like and how to measure it.

These calculations can be quite complicated and time-

consuming. Many organizations find ROI to be desirable, but

too complicated or costly to calculate. Others have

successfully implemented ROI programs.

TVA University, at the Tennessee Valley Authority, has

developed a method for calculating the ROI of each of the

hundreds of courses it offers each year. After calculating

costs through some pretty standard measures, TVA University

then calculates benefits by measuring the extent to which

each worker uses the material taught in the class in the

course of his or her job, the amount of improvement in that

area as a result of the course, and the workers value to

the organization (as measured by annual salary). Needless

to say, some of the measures are hard to get at, but the

statisticians at TVA University have devised ways of

measuring each of these elements and the university can

assess the ROI for each of its courses.

Motorola University uses human capital as the locus of

measurement at the heart of its evaluation methods (Barney,

2002). Eschewing satisfaction measures and volume measures

(such as the number of student days of training provided)

Motorola Universitys measurements focus on the value of

human capital. By analyzing performance gaps, employee

performance, and business opportunities, Motorola

University can calculate the value of closing the

performance gaps and therefore determine the ROI of

investing in improving human capital.

Organizational Metrics for Corporate University

Effectiveness

While Kirkpatricks model, ROI measures, and human

capital metrics all serve as good tools for measuring

individual programs, when it comes to evaluating a

corporate university in its entirety, we advocate a more

holistic approach. As organizations invest more and more

money in corporate universities, it becomes increasingly

important for corporate universities to justify their

existence. In order to do this, we recommend mission-based

metrics that are organization-specific.

As we mentioned in our definition, corporate

universities exist to help their parent organizations

achieve their mission. Each corporate universities should

have its own mission statement, and that mission should be

based on the specific needs of the organization. Not every

organization has a corporate university, nor does every

organization need one. But those that have chosen to

establish a corporate university presumably have done so

for a reason. The answer to the question, Why do you have

a corporate university? should drive the measurement of

the effectiveness of the corporate university. If you can

articulate an answer to that question clearly and

specifically, that should serve as a roadmap for what to

measure. And if you cannot answer that question clearly and

specifically, then measurement is not your greatest

problem.

This raises the most crucial issue in corporate

university evaluation: measurement is not something to be

undertaken after the completion of educational activities

it must be baked in up front during the planning process.

As a corporate university and its activities are being

planned, the question of why? must be asked at all times.

All corporate university activities must be goal-directed.

If you can articulate goals for each activity and for the

university as a whole, then achievement of those goals is

what should be measured. The question of what to measure is

simply the answer to the question, What were you trying to

accomplish? Whatever it is that you were trying to do is

what you measure. And the close connection between goals

and measurement reinforces the notion that measurement is a

part of the planning process, not something to be

considered after the fact.

Because of the goals-based focus of corporate

university measurement, often the evaluation process does

not look at traditional measures of learning, but rather it

focuses more on organizational goals. While learning-based

metrics are targeted at individuals, the locus of

measurement for many corporate universities is the

organization. For this reason, we often advocate that

organizational measures (such as profitability, growth,

employee retention, employee satisfaction, productivity,

and many others) are the appropriate metrics for gauging

corporate university success.

For example, the MGM Grand Hotel and Casino in Las

Vegas created its corporate university, the University of

Oz, prior to opening its doors in December of 1993. The

hotel was planning to open with 5,005 guest rooms, which

would make it the largest hotel in the world from its first

day in business. The MGM Grand needed to hire 8,000

employees by its opening day. In addition to the monumental

task of locating, interviewing, and hiring such a large

number of employees, the management at MGM knew that one of

the biggest challenges in the gaming industry is employee

turnover. There are so many large hotel-casinos in Las

Vegas that employees have a great deal of choice about

where to work and change jobs frequently. As the largest

hotel-casino in the city, the MGM Grand knew that high

turnover would be quite costly.

Early in the planning process, the management at the

MGM Grand decided to make a conscious effort to focus on

employee retention. The University of Oz was at the heart

of the retention efforts. Through a thoughtful

acculturation process, the University of Oz created a

culture where none existed before and worked hard to make

all newly hired employees feel that they were a part of

that culture.

Moreover, the University of Oz developed an internal

degree program, the Th.D. (it stands for Doctor of

Thinkology, the same degree the Wizard of Oz bestowed on

the Scarecrow at the end of The Wizard of Oz). A sequence

of courses spanning many months was required to earn the

degree. Although the Th.D. designation was quite

prestigious within the walls of the MGM Grand, it was

meaningless outside of the company, so employees who earned

the degree were less likely to change jobs. This is

another example of an employee retention effort spearheaded

by the University of Oz.

When it came time to determine if the University of Oz

was doing a good job, management did not need any complex

formulas. Since the primary focus of the University of Oz

was employee retention, that was the organizational metric

that was used as the yardstick for the University of Oz.

Since the MGM Grand had low turnover and high retention,

the University of Oz was deemed a success. While

statisticians might quibble that it is difficult to prove a

causal relationship between the efforts of the University

and the organizational results, we would argue that it is

not necessary in a case like this to prove causality. The

organization had a specific result that it wanted to

achieve, it set up a mechanism to help achieve those

results, and the ultimate goals were met. By any measure,

this is evidence of a successful outcome.

This example demonstrates several important points.

First, there is no single method that is best for corporate

university evaluation. Every corporate university exists

for different reasons, and therefore every corporate

university should have different metrics. Secondly, by

focusing on the specific mission and goals of the corporate

university, it is easy to determine what should be

measured. Moreover, it shows that evaluation need not be

complicated or costly. A highly focused mission can lead to

simple metrics.

The Productivity Model

As described earlier, Donald Kirkpatrick created a

four level of approach for evaluating training programs.

Building upon this foundation, Jack Phillips added a fifth

level, i.e., measuring the Return On Investment (ROI)

produced by training programs. However neither the

Kirkpatrick model nor Phillips ROI formula provide the

type of information needed by instructional designers and

corporate university leaders to improve the productivity of

instruction and related activities. In response for an

alternative evaluation model, McGee has developed the

Productivity Model.

To begin, let us recognize that corporate universities

value and implement a variety of instructional systems to

achieve predetermined goals and outcomes. It is possible

describe any system as being composed of three basic

elements: Resources + Activity = Results

First, let us recognize that all systems produce

results, even though the results they produce may at times

be difficult to predict. Secondly, to achieve results,

something must happen; something must be done. In other

words, activity must take place. Third, in order to have

activity, resources must be used. These three elements of a

system hold true whether the system is mechanical,

electrical, biological, financial, social or educational.

The next step to understanding Productivity Model is

to examine the dynamic relationships that exist between the

elements of a system. We will start with the relationship

between results and activity.

Effectiveness

A system is effective only when an activity creates or

produces a predetermined result with a high degree of

predictability, in other words, effectiveness is concerned

with "how well" something works. This concern for "how

well" is the basis for another concept known as quality. As

designers and managers of systems, we must strive first for

effectiveness. For without it, there is little reason to

proceed with the design or implementation of any system.

Efficiency

Efficiency is the dynamic relationship that exists

between resources and an activity. Efficiency asks the

question, "How much?" However, this is a dangerous question

to ask. It is dangerous in that if we cut resources too

much, we run the risk of producing poor results. This is

not to say that we should not be concerned with resources

and their associated costs. We should, because within every

system there is an optimum balance between resources and

activity, and activity and results. This optimum balance is

known as productivity.

Figure 1. The Productivity Model

Productivity

We can see this balance between effectiveness and

efficiency in Figure 2 below. System (A), which is neither

effective nor efficient, cannot be said to be productive.

System (B), while very effective, cannot be said to be

productive, because it is not efficient. System (C), while

efficient, is not effective. Therefore it cannot be

considered productive. Only system (D), which is both

effective and efficient, can be said to be productive. In

application, it is possible to establish metrics

(performance standards) for the system being evaluated in

terms of effectiveness and efficiency, and furthermore to

determine corrective courses of action when evaluative data

is generated and graphed.

Figure 2. Effectiveness vs. Efficiency

Applying the Model to Improving the Productivity of

Training Systems

If we are to measure and improve the productivity of a

training system, we must focus our attention on two

distinct measures. The first is the quality of the

instruction, i.e., what should be taught? And the second is

the quantity of resources needed to deliver the

instruction, i.e., by what means should the curriculum be

taught?

Figure 3. Curriculum Development and Instructional Strategy

To expand on this concept, instructional quality is

the major concern of curriculum development. Curriculum

development answers the question, What should be taught?

It is during the development of the instructional

curriculum that a performance standard for the curriculum

is established and program content is identified.

Instructional quality is often measured by how well

participants are able to predictably achieve the objectives

of the program. This data is most often gained through

traditional testing procedures. For example, a particular

curriculum may produce results, wherein an average

participant achieves a 95% level of competency on the

material presented.

Instructional quantity, on the other hand, falls into

the domain of instructional strategy, which is the process

of determining and selecting the most efficient method and

media for delivering a program of instruction (curriculum).

The goal of instructional strategy is to answer the

question, By what means should the curriculum be taught?

Again, a performance standard must be established by which

to measure this dimension. Common standards are money,

time, instructional staff, equipment required, i.e.,

instructional resources.

In order to determine the productivity of an

instructional system, we must consider both the results

produced by the curriculum and the instructional resources

required to deliver the curriculum. Keep in mind that

productivity is a ratio or composite measure of both the

effectiveness and efficiency of a system.

The power of the Productivity Model is that it enables

trainers and instructional developers to identify where

they should take corrective action.

There are a number of factors that influence the

effectiveness of an instructional system (curriculum

development). Among these are:

Needs Assessments

Assessment of Learners

Analysis of Work Settings

Job, Task, or Content Analysis

Statements of Performance Objectives

Performance Measurements (test items)

Sequence Performance Objectives

The factors that influence the efficiency of an

instructional system (instructional strategy)are:

Instructional Techniques

Designs for Instructional Materials

Instructional Resources, i.e., money, time,

instructional staff, or required equipment.

Where and When to use the Productivity Model

Because the Productivity Model is a logical construct,

it may be used to evaluate and fine-tune the development

activities within any phase of the ADDIE model, i.e.,

analysis, design, development, implementation or

evaluation. Furthermore, the Model may be used at all

levels of evaluation as described by Kirkpatrick (1994) and

used in conjunction with the concepts espoused by Phillips

(1997) and his work related to measuring the ROI of

training systems.

A Grounded Model

The Productivity Modeled described in this paper is

based upon established principles from a variety of

disciplines.

The basic elements of a system: resources, activity

and results, are described throughout the literature on

systems theory and are sometimes referred to simply as an

input/output model.

Accountants and financial people, who often speak of

return on investment, have known the relationship between

resources and results: (ROI). ROI is where returns are

results and investments are costs, and in this way, are

able to determine the health of a business enterprise

(activity).

Physicists and engineers recognized long ago that in

order to describe the performance of various phenomena and

systems, they had to be described in terms of dynamic

relationships between two variables. For example, miles per

gallon or feet per second.

Thomas Gilbert (1978), a founding father of the human

performance field, developed the First Leisurely Theorem,

which says that worth is equal to value divided by costs.

In other words, activity adds value to resources (cost) and

results in something of greater worth.

Peter Drucker (1974) pointed out that effectiveness

was doing right things, while efficiency was doing things

right. In the field of education, these concepts were

expanded upon by Ivor Davies (1981) and applied to

decision-making concerning instructional methods.

Conclusion

We illustrated that corporate universities, despite

eschewing the term institutional research, have developed

a variety of useful and creative methods for evaluation.

The menu of methodologies presented in this chapter are not

meant to be exhaustive, merely instructive. Some

organizations might choose to duplicate or adapt some of

the techniques highlighted in this chapter. If not, the

lesson to be learned is that evaluation is mission-

specific, and the thoughtful application of the techniques

in this chapter and some common sense can lead to the

development of evaluation methods that match the goals of

the parent organization, the mission of the corporate

university, and the specific informational needs that drive

the desire to evaluate.

References

Allen, M. (2002). The corporate university handbook. New York: Amacom.

Barney, M. (2002). Measuring ROI in corporate

universities: Death of the student day and birth of human capital. In M. Allen (Ed.), The corporate university handbook. New York: Amacom.

Davies, I. K. (1981). Instructional techniques. New York:

McGraw-Hill Book Company. Drucker, P. (1974). Management tasks, responsibilities,

practices. New York: Harper & Row Publishers. Eurich, N. P. (1985). Corporate classrooms. Princeton, NJ:

The Carnegie Foundation for the Advancement of Teaching.

Gilbert, T. F. (1978). Human competence. New York: McGraw-

Hill Book Company. Kirkpatrick, D. (1994). Evaluating training programs. San

Francisco, CA: Berrett-Koehler Publishers. Phillips, J.J. (1997). Return on investment in training and

performance improvement programs. Boston, MA: Butterworth-Heinemann.

Renaud-Coulon, A. (2002). Corporate universities in

Europe. In M. Allen (Ed.), The corporate university handbook. New York: Amacom.

Senge, P. M. (1990). The fifth discipline. New York:

Doubleday. Thompson, G. (2000). Unfulfilled prophecy: The evolution

of corporate colleges. The Journal of Higher Education, 71, (3), 322-341.

About the Authors

Mark Allen, Ph.D. is an author, consultant, and

educator who has specialized in corporate universities. He

is the editor and lead author of The Corporate University

Handbook, published in 2002 by Amacom. Mark is also the

Director of Executive Education at Pepperdine Universitys

Graziadio School of Business and Management, where he is

responsible for designing and delivering executive

education programs to corporate clients. He is also an

adjunct professor at the Graziadio School and at

Pepperdines Graduate School of Education and Psychology.

He lives in Redondo Beach, California with his wife Dayna

and two sons, Skyler and Dylan.

For seventeen years, Philip McGee, Ed.D. was involved

in the design and implementation of instructional and

organizational systems within education, government,

industry and business through his company, Instructional

Designs, Inc. During this time he also served as the

Regional Program Director for the Doctor of Business

Administration program offered by the School of Business

and Entrepreneurship, Nova Southeastern University, and as

an Adjunct Professor of HRD for Webster University. Dr.

McGee is currently an Assistant Professor of Technology and

HRD at Clemson University.