Paper Title: Successful people and organizational development in a service

company: the processes, results and the underpinning guidance framework.

Authors: Kesiena Mercy Clement-Okooboh and Professor Bill Olivier

Affiliation: University of Bolton

Institute for Educational Cybernetics

Corresponding author: kesienaokooboh@gmail.com

1

Abstract

Purpose

This paper presents a work-based action research case study undertaken in a national branch of a large multinational company to help move towards its strategic objective of becoming a learning organization.

Design/methodology/approach

It describes how training programs in the organisation were enhanced through a series of organisational processes and practices to become integrated with workplace learning. This paper draws on the evaluation and learning organization literature to present the linkages made in practice between the training process, procedures and the business goals.

Findings

Findings in the literature identified the varying impact of training on different stakeholders and these were used to integrate higher levels of evaluation into a work embedded feedback loop not only to improve training programs, but to transition towards a learning organization that is adaptive to its changing operating environment.

Social/Practical implications

This case study’s value lies in its proposal of a guidance framework generalised from the concrete instance of implementing post-training performance support, that can be used to enable the creation of a learning organization, a task that has been found difficult to do in practice.

Originality/value

This case study’s value presents a widespread perspective in providing a concrete instance of adopting training to enhance performance in the organization thus enabling the creation of a learning organization, a task found difficult to do in practice.

Keywords:

Learning organization; higher-level evaluation; training evaluation; feedback loop;

training; informal learning; work-based learning

Article Classification: Working Paper

(239 words, including keywords and Classification)

2

Introduction

This work-based action research study presents an overview of the organizational

context, the aims and objectives, the research questions and action research cycles that

constituted my work-focused PhD. It explores how this study identified four

approaches to successful people and organizational development in a service industry.

The first needs based approach centers on meeting individual, unit and organisational

needs, whilst delivering immediate and future benefits to our clients.

The second performance enhancement approach focuses on the need to enhance

organisational performance of individuals and teams through transfer of knowledge,

skills and attitudes gained from training programs to the workplace. The measure of

success of a training program lies in determining the changes in work place behavior,

unit performance enhancement, and customer satisfaction.

The third informal learning approach introduces informal workplace learning, using

the 70:20:10 framework and performance support into the work-based.

Finally, the fourth evaluation and feedback approach involves summarizing and

communicating the findings and results of the evaluation to both internal and external

stakeholders, to establish the benefits of what has been learned and what further

learning is then needed.

In summary, the needs based approach drives the formal and informal learning;

performance enhancement seeks to ensure this is carried through into practice;

evaluation establishes the extent to which this has happened; the evaluation results

feed into further learning and improved performance.

Organizational Context

The context of this case study is in an energy & utilities company, based in Dublin

operating across the Republic of Ireland and Northern Ireland, with over 350

employees. Its parent company is a large French multinational with operations in 42

countries. The company provides solutions to support the sustainable growth of cities

and businesses. Two-thirds of its’ workforces are technicians and engineers.

Aims and Objectives

At the outset of this study then, there were two major, open and interrelated questions

facing the company:

1. How to achieve its strategic aim of becoming a “Learning Organization”

3

2. How to achieve the “Engineers Ireland CPD Accreditation”

The goal of this work-focused action research project was then to facilitate the

strategic transition of this company from a training-based one towards a full learning

organization.

Background of the study

The study stemmed from the evaluation of three training courses, namely supervisory

management, certified energy management and boiler operative accreditation scheme

in the organisation. The reason for selecting only three training courses to be

evaluated at these higher levels was because it is not practical to evaluate every course

at these levels. These three courses were deemed ideal for evaluating at the results

level of the Kirkpatrick model because they were expensive, strategic and

operationally focused (Phillips, 2007). According to Phillips (2007) he asserts, “Only

a few select programs should be the subject of evaluation at the ROI level”. The

purpose of the study was to address the evaluation of its training programs to achieve

a sustained performance improvement of its process that adds value to the

organisation. Evaluation at the different levels is an important influence in training.

For performance improvement to be achieved, learning and development practitioners

must first identify the reason for evaluating training that ultimately leads to a high

performance culture. This led to the focus of the four approaches that included other

factors like mentoring, informal learning and performance support aimed at achieving

sustained performance improvement.

The Four Approaches

An important issue informed the four approaches. For the evaluation of training to be

part of a dynamic process in which “sustained performance results” can be achieved it

has to be accepted as an organisational challenge (Brinkerhoff, 2005).

As well as ensuring that appropriate training is provided, the critical evaluation

measures employed in all four approaches also enabled the impact of the training on

performance of employees and the work unit to be enhanced through additional

workplace measures. Evaluation was also extended to include informal work-based

learning in the evaluation and feedback loop, thus moving from the evaluation of

training, to the evaluation of learning as a whole.

4

First Approach: Needs based

The first approach required getting line management involvement in the learning

process. The mechanisms put in place included line managers and employees

attending pre-training meetings where a unit’s operational goals and the employees

corresponding learning needs are established. The first step was to identify the needs

base, as this drives both the required training and the evaluation process, as the needs

of the different stakeholders involved provide the evaluation criteria.

Training measurement and evaluation can be a painstaking task for HRD

professionals. The aim of measurement and evaluation is to continuously improve the

training interventions. The need to determine the impact of training programs led to a

review of the training evaluation literature. It became clear that the Kirkpatrick

Training Evaluation method (Kirkpatrick, 1959) was the most widespread in use, and

that a number of others were variations on this approach (Hamblin, 1974; Phillips,

1996; Holton, 2005; Brinkerhoff, 2003). To evaluate both individual and unit

performance, it was necessary to establish both the unit’s unmet or insufficiently met

goals, the tasks it would need to perform to achieve them, and the corresponding

needs of the training delegates would be in order to perform the required tasks. In

establishing a process for making effective use of the evaluation data being gathered,

it became clear that their primary function was to provide a feedback loop both to

help assess whether the goals, as set out in the pre-course discussion meetings, have

been met, and to enhance the training and learning processes themselves. In

attempting to examine the impact of our learning and organisational development

practices, a critical early finding from the evaluation of a selection of our training

programmes was that while 80 per cent of training delegates could successfully repeat

what they had learned, relatively few, approximately 20 per cent, were able to

translate this into changed performance in the workplace. This confirms a

corresponding gap, identified in the literature, between the learning gained from the

training, and its transfer to the workplace (Baldwin and Ford, 1988, p.63; Detterman

and Sternberg, 1993; Saks and Belcourt, 2006). An important point of debate in the

literature indicates a general lack of higher-level evaluation. Kirkpatrick and

Kirkpatrick (2006) highlights that “Evaluating results, Level four, provides the

greatest challenge to training professional” (p. 69). Kirkpatrick further (2007)

reiterated, “the four levels are well known and poorly implemented” (p, 4). There is a

consensus in the literature on the evaluation challenges. According to Sugrue and

5

Rivera (2005) their ASTD review of trends in workplace learning and performance,

training evaluations occur in organizations at the following percentages: level one

(reaction of learners) 91%; level two (learner knowledge) 54%; level three (transfer of

learning on the job) 23%; level four (impact of learning on the business) 8%; and

level five (return on investment) 3%. The reason for the low evaluation rate at the

higher levels can be linked to the findings of Devins and Smith (2013). They argue

that line managers perception of evaluating at higher levels may seem too time-

consuming and of minimal interest. According to Brinkerhoff (2006), “When

managers support training and learners, it works. When they do not, it does not.”

Given these findings, in order to address the issue of taking this less travelled full

evaluation route, it is essential to begin with a description of the purpose of evaluating

learning programs and their benefits to organizations. Many studies in training

evaluation have shown the need for measurement and evaluation both to account for

training investment and to drive organizational results. In order to convince

organizational leaders that learning programs are effective, the need for accountability

has led learning professionals to embark on measurement and evaluation of

organizations training. This feedback process comprises both qualitative and

quantitative as well as financial and non-financial data (Philips, 2010).

Second Approach: Performance Enhancement

The second approach identifies the aspects of the evaluation process that feeds into

improved performance. Learning and development programs incorporate a huge

investment from a company’s perspective (Calhoun et al, 2006). To thrive in their

competitive ventures, organisation leaders must be able to change and adapt to the

marketplace (Calhoun et al, 2006). To show how their strategies contribute to

improving business results, learning and development professionals need to

demonstrate that they own the process before, during and after the learning initiatives

employed (Calhoun et al, 2006). Achieving this purpose is to define their programs’

objectives in business terms.

It is well established that half of corporate expenditure on training is wasted, because

the training of employees is undertaken far too long before the first opportunity

trainees have to apply the skills and knowledge on the job (Cross, 2007). Kirwan

(2009; 2013) found that between1% and 4% of payroll cost is spent on formal

learning and development in Ireland, but only between 34% of this investment is

6

applied on the job by employees after training (Saks and Belcourt, 2006). Taking into

consideration the complexity of evaluation in an organizational context, a systems

thinking approach was adopted to determine the other organizational factors that may

have both contributed to the results and may have also themselves been accelerated

through the learning transfer and change in behavior. To foster best practices for

learning effectiveness within organizations, which ultimately leads to performance

enhancement, organizations need to carry out training evaluation (Kraiger et al., 2004;

Collins, 2002). In other to determine the effectiveness of learning and development

practices, measurements of these practices must be conducted. Training outcomes are

determined by a combination of mechanisms that influence how people process

information, focus their attention and direct their effort during learning (Ackerman &

Kanfer, 2004; Ackerman et al, 1995; Kanfer et al, 1996; Kozlowski, Toney, et al.,

2001). According to Robinson (2007), “training is only one method of improving an

organizations performance”. However, Cross (2010), in his summary of learning in

the 21st century workplace, demonstrates convincingly that “work and learning are

converging into the new conceptual work” This tells us that work and learning cannot

be separated from each other. This interdependence has given rise to informal

learning and the 70:20:10 framework that will be discussed in the third approach.

Developing and measuring the delegates’ skills depends on other inherent

organisational factors and initiatives namely, performance support, mentoring,

informal learning, and other initiatives like development plans, and their associated

measurement practices, and these constituted a major part of the transition program.

Third Approach: Informal work based learning

The evaluation of learning, particularly at the higher levels of individual, unit and

organisational performance, is important as it informs the work based learning

approach. This in turn is a key aspect of our people development approach as it puts

the responsibility for learning primarily in the hands of the units and the learners. This

approach led to the adoption of the 70:20:10 concept Eichinger and Lombardo,

(1996), which is based on findings that, on average, 70 percent of all learning within

an organisation is informal, and happens on the job, 20 percent from semi-formal peer

review, feedback, mentoring and coaching and only 10 percent of learning is formal

(Australian Industry Group, 2006; Cross, 2006; Jennings and Wargnier, 2010; Meister

and Willyerd, 2010). Simply raising awareness of this concept within the organisation

7

had a significant impact on attitudes towards learning. This approach led to the

introduction of performance support mechanisms that ensured line managers and

delegates planned the support that will be available after the training. The support

provided by in-house mentors alongside the level four evaluation of training

programme, facilitated the translation of performance results into organisational

benefits. The performance support mechanisms provided a good platform for the

development of informal workplace learning. The relationship between training and

performance enhancement systems in organisations’ is one of mutual

interdependence. This is taking an important role in the education and training of the

workforce (Johnston and Hawke, 2002).

Taking this philosophy of workplace learning in the workplace at the outset of this

study. First, employees were of the opinion that they only learn when they attend

formal classroom training. Second, employees were not taking responsibility for their

own learning. Third the organisation wanted to record all learning in a central hub in

order to carry out analytics and to facilitate referencing and sharing of key

information with the senior management team.

Our intention in this study was to create a sustainable learning process that

incorporates informal workplace learning that translates into change in behavior and

performance improvement of individuals and business units in the organisation.

Fourth Approach: Evaluation feedback

Once all three approaches were put in place, the focus of the fourth approach was to

communicate all of these findings and results of the evaluation to the wider

organization. Sloman (2003) advocates the communication of learning and

development practices across the organisation. This programme adopted and adapted

the Kirkpatrick four–phase framework for evaluating learning effectiveness and

builds on its evaluation taxonomy, namely 1. Reaction to the learning event, 2. The

learning as tested verbally or in writing after the event, 3. behavior change (or

individual performance) and Results (or unit/organisational performance). The

adoption of a cognitive approach by organizational theorists has identified a number

of principles that encourage and enhance learning, such as motivation. The objective

of learning is to change the experience by impacting behavior and cognition

(Williams, 1998), to improve performance and to successfully transfer the learning to

the workplace (Clifford and Thorpe, 2007). Senge (2006) argues that the most

8

powerful learning comes from direct experience. Once the results from the training

programs and semi-structured interview were analysed and collated, the findings

were shared with the different stakeholders – individuals, line managers, clients’,

trainers and the senior management team via CPD meetings, the monthly Learning

Talk newsletter and workshops.

The Underpinning Guidance Framework

To help communicate the whole process to colleagues and to make it more usable by

others, both with in the organisation and externally, a guidance framework was

developed which is outlined in the following overview diagram and the

accompanying description.

Figure 1: Overview of the unit level learning framework

The new process starts (top left in the diagram) with managers and training delegates

attending a pre-training meeting, where the unit’s operational needs and the training

delegates’ corresponding learning needs are agreed. These then inform those who are

providing the training/mentoring/performance support. Critically, they are also used to

provide the criteria for the follow-on personal and unit performance evaluations. The

training or other learning event then takes place and is initially evaluated at level 1, the

9

learners’ reactions to the event, and level 2, the test of what has been learned. After the

learning event, the delegates should begin to change how they perform their tasks, and

this in turn should bring about changes in the unit’s operations, helping it to better

achieve its goals. Later, perhaps some months later, level 3, the actual changes in the

delegates’ task performance, and level 4, the unit’s operational performance, are also

evaluated. The collected results of the evaluations are provided as feedback to the post-

training meeting of the managers and training delegates and compared with the original

aims. This review then provides further feedback to the trainers, mentors and those

providing workplace support. New or unmet goals and needs are then considered and

become inputs to future learning events and evaluations. Improvements are also

proposed for the unit’s internal mentoring, performance support and other informal

learning activities.

The framework, initially developed around training, was later extended to include

performance support, mentoring and informal learning, and, after the first few

iterations, the pre- and post-course meetings were merged into unit learning meetings.

These are held at regular intervals and review the unit’s goals, the learning that has

taken place, and what learning is still needed.

An important aspect of these unit learning meetings is that they put the responsibility

for learning primarily in the hands of the operational units and the learners. They set

the goals, which inform the evaluation, which then provides them with the feedback

needed to assess and improve their learning.

However, the limitations on generality inherent in a specific case study are

acknowledged and the framework is offered to others wishing to follow a similar path

towards establishing a learning organisation, in the hope that it will prove useful and

that it will become clear whether and how if so how much it needs to be adapted to fit

different circumstances.

Research Methodology

This programme adopted both qualitative and quantitative methods, use of

questionnaires, observation and semi-structured interviews. The analyses were

conducted on twenty-three delegates that attended three different training courses. We

10

employed data from three different questionnaires that assigned a range of

instruments namely:

Pre-course questionnaire – Before commencing the training, this questionnaire

was sent to both the delegates and the line managers to set the goals and

objectives of the training, the needs of the individual, team and manager are

outlined.

Level one evaluation questionnaire – This questionnaire employed likert scale

and open-ended questions to gather findings on the delegates perception of the

training courses, the trainers facilitation skills and the overall programme. This

questionnaire is administered at the end of the training.

Level three-evaluation questionnaire – This post-course questionnaire is sent

to the delegates six to nine months after completing the training course to

determine if the delegates have changed their behavior and applied the new

knowledge and skills in the workplace to enhance their performance.

Semi-structured interviews (Yin, 1989) were conducted with line managers of

the training delegates to assess the impact of the training on the unit’s

performance. The interviews were carried out six to nine month’s period after

the training.

Line managers and also the trainer conducted observation of the delegates

during and after the course.

The data from each of the instruments employed was analysed and compared to each

other to determine the factors that may have achieved the performance results from

the training. The evidence from this case study includes comparing the goals and

objectives set during the pre-training meetings and the outcomes during the post-

training meetings. Once the results from these meetings were determined, a review of

the other factors that may have contributed to the performance improvement results of

the training courses was established through the semi-structured interviews with the

line managers. In the interviews questions were asked by the line managers of the

delegates how the delegates have applied the new skills/knowledge to their job. This

question was asked to gain insight in the application of learning and development

process. Next, questions were asked about the impact of training on the different

stakeholders.

Findings and Discussion

11

In the first approach, it became apparent that there was a need to carry out a pre-

course discussion between delegates and their line managers to set the goals and

outcome before the training commenced. It was also determined during pre-course

discussions that there should be post-training discussions/ meetings of the delegates

and their line managers with the L&D Unit to determine if the pre-course goals had

been achieved. Aspects of the second approach on the rate of training transfer were

investigated, had been carried out prior to the commencement of this study as well as

during it. The results of the prior evaluation showed that most of the delegates

sampled did not have a clear understanding of the objectives of the specified training,

or even why they attended. Some also mentioned that they did not have the

opportunity to apply the new knowledge and skills back in the workplace. Line

managers also felt that allowing the delegates to attend the training programme would

increase their morale in the team, where it had been identified that the training

delegates were demotivated. The revealed lack of successful transfer from the training

to the workplace was a concern to all.

Subsequently, when delegates were surveyed after these new processes and

procedures were implemented in the organisation, the rate of transfer from learning

changed performance in the workplace. This was found to have increased

significantly from 20% to 95%. It also became apparent that the pre-course discussion

had encouraged both delegates and line managers to plan performance support

mechanisms to put in place after the training. The introduced changes thus had a huge

impact on the individual, team and organisational performance. In the third approach,

the development of an interrelated strategy encompassing performance support,

informal learning, and the introduction of the 70:20:10 concept and the in-house

mentoring programme helped to strengthen and integrate with the organisational

learning strategy. This informal learning part of the programme was deemed a critical

factor as the line managers now set out work related activities that gave the training

delegates the opportunity to apply and practice what they had learned, thus not just

preventing loss of the new knowledge and skills, but establishing them more firmly.

The goal of higher-level evaluation is to encourage better ways of ensuring the change

of behavior of the delegates that attended training programs. The output of the

training had a higher effectiveness rate and this was of importance to the organisation.

The final approach highlighted the summary and communication of the different

processes to the different stakeholders across the organisation. Using this approach

12

ensured that these new processes are understood by all to drive a process of

continuous improvement. The main aim of adopting this strategy was to enhance the

adoption of the processes to ensure its effectiveness in achieving the business results.

During the semi-structured interview process, nine line managers were interviewed as

part of this study. Firstly, they were interviewed to determine if the delegates had

changed their behavior or performance subsequent to the training. Secondly, they

were interviewed to determine how the training had impacted on the unit, clients or

business. It was confirmed that evaluating performance at the levels of individual and

unit/organisational performance is more difficult than the levels of reaction and

learning evaluation. As a result, relatively few organizations evaluate the impact of

training (Brinkerhoff and Gill, 1994; Rothwell and Benkowski, 2002). Clearly, the

line managers play an integral and critical role in facilitating knowledge transfer as, in

addition to performance support, they need to provide the right conditions, within a

reasonable timeframe, for delegates to develop the use of their new skills on the job

(Gottfredson and Mosher, 2011). Furthermore, the learning and development unit

established a process for making effective use of the evaluation data being gathered.

The data enables improvement of the training courses and also to inform what needs

to happen in the workplace. As a result, promoting evaluation at these higher levels

enabled the organisation to move from viewing learning as something that happens in

the classroom to something that mainly happens in the workplace. This indicates that,

both learning in the classroom and the workplace are complimentary.

Conclusion

As with any HRD initiatives, the transfer of learning back to the workplace has been

the challenge for HRD professionals. The responsibility of assessing, measuring and

evaluating of any HRD activities relies solely on processes and procedures inherent in

the organization. The results from this study highlight strong links between our

approach to people development and the inter-woven goal of single feedback loop of

the new processes, procedures and frameworks introduced in the organisation. The

focus for us was to align our people development practices closely to enhance the

performance of training delegates. This has enabled employees to pull information

and learning from across the organisation into a positive, productive and supported

learning transfer in the workplace. Finally, this paper sets out how all of these

practices are now considered together with formal learning in the learning reviews.

13

Reference:

Ackerman, P.L. and Kanfer, R. (2004). Cognitive, affective, and conative

aspects of adult intellect within a typical and maximal performance

framework. In D.Y. Dai and R.J. Stenberg (Eds.), Motivation, emotion, and

cognition: Integrated perspectives on intellectual functioning (pp. 119-141).

Mahwah, NJ: Lawrence Erlbaum Associates.

Ackerman, P.L., Kanfer, R., and Goff, M. (1995). Cognitive and non-cognitive

determinants and consequences of complex skill acquisition. Journal of

Experimental Psychology: Applied, 1, 270-304

Australian Industry Group (2006); World Class Skills for World Class

Industries: Employers Perspectives on Skilling in Australia. Australian

Industry Group, Sydney

Baldwin, T. T., and Ford, S. K. (1988). Transfer of training: A review and

directions for future research, Personnel Psychology, 41, (1) 63-105

Brinkerhoff, R.O and Gill, S.J. (1994). The Learning Alliance: Systems

Thinking in Human Resource Development, Wiley, Chichester

Brinkerhoff, R. O. (2003). The success case method. Find out quickly what’s

working and what’s not. San Francisco: Berrett-Koehler.

Brinkerhoff, R.O. (2005). The success case method. A strategic evaluation

approach to increasing the value and effect of training. In G.G. Wang and D.R.

Spitzer (Eds.), Advances in HRD measurement and evaluation: Theory and

practice. Thousand Oaks, CA: Sage

Brinkerhoff, R.O. (2006). Telling training’s story: Evaluation made simple,

credible, and effective. San Francisco: Berrett-Koehler

14

Calhoun, W. et al (2006) The Six Disciplines of Breakthrough Learning: How

to Turn Training and Development into Business Results. Library of Congress

Cataloging-in-Publication Data.

Clifford, J. and Thorpe, S., (2007) Workplace Learning & Development.

Delivering Competitive Advantage for your Organizations. Kogan Page.

London

Collins, M.E. (2008). Evaluating child welfare training in public agencies:

status and prospects. Evaluation and Program Planning, 31, 241-246

Cross, J. (2006) Informal Learning: Rediscovering the Natural Pathways That

Inspire Innovation and Performance. Pfeiffer Publication

Cross, J. (2007) Informal Learning. Rediscovering the Natural Pathways That

Inspire Innovation and Performance. Pfeiffer Publication.

Detterman, D.K. and Sternberg R.J (1993), Transfer on trail: Intelligence,

cognition, and instruction. Ablex Publishing, New York, NY

Devins, D. and Smith, J. (2013). Chapter 8 Evaluation of HRD, in God, J.

Holden R., Stewart, J., Iles P., Beardwell, J. Ed (2013) Human Resource

Management, Theory and Practice, 2nd edn. Palgrave MacMillian, Basingstoke

Eichinger, R. and Lombardo, M. (1996) The Career Architect Development

Planner: A Systematic Approach to Development Including 103 Research-

based and Experience-tested Development Plans and Coaching Tips: for

Learners, Managers, Mentors, and Feedback Givers. Lominger Limited,

Minneapolis

Ford, J.K. and Weissbein, D.A. (1997). Transfer of training: An update review

and analysis. Per

Gottfredson, C., and Mosher, B., (2011). Innovative Performance Support:

Strategies and Practices for Learning in the Workflow. McGraw Hills

Companies

Groat, L. and Wang, D. (2002). Architectural research methods. New York.

John Wiley

Hamblin, A. (1974), Evaluation and Control of Training, McGraw-Hill,

London

15

Holton, E. F., III. (2005). Holton’s evaluation model: New evidence and

construct elaborations. Advances in Developing Human Resources, 7(1), 37-

54.

Jennings, C. and Wargnier, J. (2010) “Effective Learning with 70:20:10: The

new frontier for the extended enterprise. Available at:

www.crossknowledge.com/ (accessed 6 July 2014)

Johnston, R. and Hawke, G. (2002), Case Studies of Organizations with

Established Learning Cultures, NCVER, Adelaide.

Kanfer, F.H., Reinecker, H., and Schmelzer, D. (1996). Selbstmanagement-

therapie. Ein lehrbuch fur die klinische praxis [Self-management therapy. A

training guide for clinical practice]. Berlin: Springer.

Kew, J. and Stredwick, J. (2013) Human Resource Management in a Business

Context. United Kingdom: Chartered Institute of Personnel & Development

Kirkpatrick, D.L. (1959). Techniques for evaluating training programs.

Journal of the American Society for Training Directors, 13(11), 3-9.

Kirkpatrick, D.L. (1996). Great Ideas Revisited. Training & Development,

50(1), 54-59.

Kirkpatrick, D.L., and Kirkpatrick, J.D. (2006). Evaluating training programs:

the four levels (3rd ed.). San Francisco, US: Berrett-Koehler.

Kozlowski, S.W.J., Toney R.J., Mullins, M.E., Weissbein, D.A., Brown, K.G.,

and Bell, B.S. (2001) Developing adaptability: A theory for the design of

integrated-embedded training systems. In E. Salas (Ed.) Advances in human

performance and cognitive engineering research (Vol. 1. Pp. 59-123).

Amsterdam: JAI/Elsevier Science.

Kraiger, K., McLinden, D., and Casper, W.J. (2004). Collaborative planning

for training impact. Human Resource Management, 43(4), 337-351

Kraiger, K. Salas E., Cannon-Bowers, J.A. (1995). Measuring knowledge

organisation as a method for assessing learning during training. Human

Factors, 37, 804-816.

16

Meister, J.C., and Willyerd, K. (2010) The 2020 Workplace: How innovative

companies attract, develop and keep tomorrows employees today. Harper

Collins Publishers

Phillips, J.J. (1996) Accountability in Human Resource Management.

Houston, Texas. Gulf Publishing

Phillips, J.J. (1996). Measuring ROI. The fifth level of evaluation. Technical

and Skills Training, 4, 11-13

Phillips, J.J. (1997a). Handbook of training evaluation and measurement

methods (3rd ed.). Boston: Butterworth-Heinemann

Phillips, J.J. (1997c). Return on investment in training and performance

improvement programs. Boston, MA: Butterworth-Heinemann.

Phillips, J.J., and Phillips, P.P (2010). Measuring for Success: What CEOs

Really Think about Learning Investments. Alexandria, VA: ASTD Press

Robin, P.L (2007). A Level 4 Evaluation of a structured on-the-job operator

training intervention at a North American Paper Mill: A Dissertation

presented in partial fulfillment of the requirements for the degree of Doctor of

Philosophy. Capella University

Rothwell, W. and Benkowski, J. (2002). Building Effective Technical

Training: How to Develop Hard Skills Within Organizations, Pfeiffer, San

Francisco, CA

Rummler, G.A., and Brache, A.P, (1995). Improving performance: How to

manage the whit space on the organization chart (2nd ed.) San Francisco:

Berrett-Koehler

Saks, A., and Belcourt, M. (2006). An investigation of training activities and

transfer of training in organizations. Human Resource Management, 45(4),

629-648

Senge, P.M., (2006) The Fifth Discipline, The Art & Practice of The Learning

Organisation, London, Random House

Sloman, M., (2003) Training in the Age of the learner: Chartered Institute of

Personnel Development (CIPD) British Library Cataloguing, Great Britain

17

Sugrue, B., and Rivera, R.J. (2005). State of the industry: ASTD’s annual

review of trends in workplace learning and performance. Alexandria, VA:

ASTD.

Tashakkori, A. and Teddlie, C. (2003). Handbook of Mixed Methods in Social

& Behavioral Research. Sage Publications

Tessmer, M., and Richey, R. (1997). The role of context in learning and

instructional design. Educational Technology, Research, and Development,

45(3), 85-115

Yin, R.K. (1989) Case Study Research, Sage: London

18