dev and delivery of industry led proj mgt project management education

8/18/2019 Dev and Delivery of Industry Led Proj Mgt Project Management Education

1/15

The development and delivery of an industry led projectmanagement professional development programme: A case study

in project management education and success management

Mehmood Alam a,*, Andrew Gale a, Mike Brown b, Callum Kidd a

a Engineering Project Management Research Group, School of Mechanical, Aerospace and Civil Engineering,

The University of Manchester, P.O. Box 88, Sackville Street, Manchester M60 1QD, UK b Programme Management Centre, Rolls-Royce plc, Derby, UK

Received 13 December 2007; accepted 18 December 2007

Abstract

Global changes influence the project environment, client relationships and the behaviour of suppliers. The people managing projects(the project management community of practice) are increasingly important, requiring professional development and training. Projectmanagement education is time and resource intensive.

Historically conventional return on investment criteria have not been applied to investment in the areas of education and train-ing. This paper reviews a case study modular distance learning programme: The Project Management Professional DevelopmentProgramme; providing education in generic project management for a consortium of four international companies across aerospace,infrastructure, oil and gas, nuclear design, construction and information technology sectors. The programme started in May 2000and has currently circa 200 delegates having graduated 100. It is an academic-industrial collaboration between The University of

Manchester (UoM) and Rolls-Royce, AMEC, Goodrich and EDS with some guest organisations participating also. The literatureon educational issues, professional development, competence and Benefit Metrics (return on training investment) are reviewed. Par-ticular attention is paid to the management and development of the programme and the project management of project managementeducation. The drivers, development and implementation of a managed learning environment, and blended learning are discussed.This includes issues related to expectation management and the interesting benefits of educating different members of the supplychain represented by the industrial partners.

The linkages between Benefit Metrics, project management competencies and learning outcomes in the context of an industrial-aca-demic partnership are specifically explored. Discussions and conclusions focus on lessons learnt and suggestions on the development anddelivery of the programme and its effectiveness. 2007 Elsevier Ltd and IPMA. All rights reserved.

Keywords: Competence; Project management; Professional development; Benefit metrics; Management development

1. Introduction

This paper concerns a modular distance-delivered gen-eric project management professional development pro-gramme for cross-sector industrial partners. It is run as aproject and the ongoing research reported here arises froma maturing academic-industrial educational partnershipwhich has been running for nearly eight years. Over that

0263-7863/$30.00 2007 Elsevier Ltd and IPMA. All rights reserved.

doi:10.1016/j.ijproman.2007.12.005

* Corresponding author. Tel.: +44 (0) 161 306 2604; fax: +44 (0) 161 3064252; mobile: +44 789 272 2122.

E-mail address: [email protected](M. Alam).

www.elsevier.com/locate/ijproman

Available online at www.sciencedirect.com

International Journal of Project Management 26 (2008) 223–237

mailto:[email protected]:[email protected]


2/15

time 100 students (delegates) have graduated with a mas-ters degree in Project Management as well as others withpostgraduate modules, certificates and diplomas.

This paper begins with a brief introduction to the discus-sion on the drivers and criteria for project success. Theauthors then move on to explain the concepts of Benefit

Metrics and Return on Investment in relation to profes-sional development education and training. Anecdotal evi-dence from industrialists suggests that companies fail totake project management education seriously because of the difficulties in demonstrating its effects on profitabilityand competitiveness. Companies are unaware of the vari-ables that influence such programmes and as a result areunable to measure a return on their investment.

The paper briefly describes the case study course: TheProject Management Professional Development Pro-gramme (a modular distance learning course) and thenreports on the interim findings from three surveys, formingthe initial phase of an industry led action research initiative

to investigate Benefits Metrics. Conclusions are drawnrelating to programme effectiveness and lessons learnedto date from the industry-academic partnership.

2. Using project management for achieving project success

In a changing and uncertain world project managementis becoming increasingly important for the delivery of suc-cessful projects and is acknowledged as more effective thantraditional functional management [1,2] in doing this. TheProject Management Institute Project Management Bodyof Knowledge (PMBoK) Guide [3] defines a project as ‘‘a

temporary endeavour undertaken to create a unique prod-uct or service”, and states that although projects vary con-siderably in type and scale they are a tool employed by theorganisation to achieve the strategic plan. Project Manage-ment on the other hand is ‘‘the application of knowledge,skills, tools and techniques to project activities to meet project requirements” [3]. Processes used in Project Manage-ment include initiating, planning, executing, controllingand closing. The Association for Project ManagementBody of Knowledge [4] takes a slightly different view, stat-ing that Project Management is the ‘‘most effective way of introducing unique change” and constitutes the successfulmanagement of projects.

These definitions are both heavily oriented to achieveproject success [2]. There is a considerable volume of litera-ture in the field of project management dealing with projectsuccess, and this tends to fall into three major categories:dealing with project success criteria; project success factorsand those that confuse the two [5]. The Iron Triangle [6]comprises three well recognised criteria (cost, time andquality) against which project success is measured (Fig. 1).

In addition, many studies have expanded project successcriteria to include such things as organisational objectives,stakeholder satisfaction, customer benefits and futurepotential for the organisation. Researchers do not reach a

consensus on project success criteria. Morris and Hough

[7] use project function, project management and the con-tractor’s business performance to measure project success.Lim and Mohamed [8] define micro and macro criteria tomeasure project success. Their micro criteria comprise

time, cost, quality, performance and safety, whilst macrocriteria include the micro criteria plus the project outcomebenefit. On the other hand some consider project efficiency,customer benefits, organisational success and the futurepotential to an organisation as critical when measuringproject success [9]. However, the majority of research prac-titioners [10–13] consider project success as an importantproject management issue [5]. For instance, the PMBoKguide published by the PMI suggests that project successcriteria should include the ‘‘Iron Triangle” and key projectstakeholder satisfaction [14].

Considering the emerging body of research on project

success, Crawford [5] identifies twenty four success factorsas primary for successful projects. The majority of thesefactors are directly related to project management compe-tence and demonstrates that the competence, knowledge,skills and attributes of project managers, are critical to project success [5]. The competence of project managers is initself a factor in the successful delivery of projects. Projectmanagers need to have competence in those areas that havethe most impact on successful outcomes.

From an industry perspective, it would be useful to beable to say that if an outstanding project manager isresponsible for a project, it guarantees that the project willbe a success, but this is not always true and whilst a poorproject manager may doom the project to failure an out-standing project manager may not necessarily guaranteesuccess. Thus, it appears that the direct causal relationshipbetween an outstanding individual and project success is, atbest, tenuous. This may be due, for instance, to the com-plexity of the project environment. Contributing factorsinclude:

(a) the quality of the project team over which, in a matrixorganisation, the project manager has little selectioncontrol;

(b) the project management maturity of the organisation;

(c) the level of stakeholder convergence;

Cost

TimeQuality

Fig. 1. The Iron Triangle [6].

224 M. Alam et al. / International Journal of Project Management 26 (2008) 223–237


3/15

(d) context stability, including technology maturity andexperience of similar projects.

It would be reasonable to suggest that knowledgeable,experienced project managers with good leadership skillsmay make a difference. Therefore, providing project man-

agement education and professional development for project managers demonstrating appropriate behavioursshould result in improved competence. However, corporatesponsorship for project management education pro-grammes has to date been a leap of faith.

3. Project management education: The concept of return on

investment

Organisations are spending significant amounts annu-ally on professional development for their employees witha view to reaping the reward in the form of competitiveadvantage. According to a training survey [15], employers

‘‘spend over $50 billion per year on formal employee train-ing and education and approximately $180 billion per yearis spent on informal, on-the-job training”. Despite thisinvestment in educational programmes in the field of project management education, it is difficult to be sure thatthe knowledge gained is linked to the context of profes-sional practice which will benefit the employing organisa-tion [16].

Expenditure on training and education represent aninvestment for which there would be the expectation of areturn deemed beneficial to the organisation. It would beexpected that Return on Investment (ROI) would be a fun-

damental consideration.The value of professional development and training pro-

grammes delivered by universities and other providers con-tinues to attract much debate. According to Jim Hinds(Marakon Associates) who advises FTSE 100 companieson the drivers of performance [17]:

‘‘Companies have failed to take training (and education)

as seriously as they should because it is hard to demonstrate

exactly how much impact it has on the bottom line and you

cannot measure it, it is hard to communicate the need for it

to the board or CEO.”There are several issues driving increased interest in,

and application of, ROI including pressure from clientsand senior managers. Competitive economic pressurescan cause intense scrutiny of all capital expenditure,including training and development costs. Engineeringprocesses such as Total Quality Management (TQM),Business Process Re-engineering and Continuous ProcessImprovement have created a renewed interest in measure-ment and evaluation, including the measurement of train-ing effectiveness. Also, the general trend towardsaccountability in organisations on functions playing a sig-nificant role in delivering projects has led to project man-agement and human resources to measure theircontribution [18].

Higher education institutions (HEIs) like The Universityof Manchester, as providers, are also interested in evaluat-ing the extent to which their programmes are successful inmeeting organisational needs. These needs include: raisingthe profile of project management, career development,promoting a cadre of project management, specific interestgroups and increasing the project management knowledgeof employees. These (or similar) factors have created anincrease in the application of ROI in project organisations.Departing from the Kirkpatrick 4 level evaluation model[19], Phillips [18] proposes a ROI model. This model(Fig. 2) comprises five levels; each investigating different

elements of a professional development/trainingprogramme.

Level 1, Reactions: measures programme participant sat-isfaction along with their plans to apply what they havelearned. Whilst this level also investigates customer

Planning

Developevaluationplans and

Baseline data

ReactionsLevel I

LearningLevel II

ApplicationLevel III

Business ImpactLevel IV

Convert Data toMonetary value

ROI levelLevel V

TabulateProgramme

costs

IdentifyIntangiblebenefits

Data Analysis

Reachconclusions/

generate report

Reporting

Fig. 2. The ROI Model [18].

M. Alam et al. / International Journal of Project Management 26 (2008) 223–237 225


4/15

satisfaction, a favourable reaction does not ensure thatparticipants have learned new skills or knowledge.Level 2, Learning: focuses on what participants havelearned during the programme, using tests, skill prac-tices, role plays, simulations, group evaluations, andother assessment tools. However, a positive measure at

this level is no guarantee that what is learned will beapplied on the job.Level 3, Application and Implementation: determineswhether participants applied what they learned on the job. Appraisal by line managers and 360appraisals arewell recognised tools for tracking the application of knowledge. Though level 3 is important to gauge thesuccess of the application of a programme, it still doesnot guarantee that there will be a positive businessimpact on the organisation.Level 4, Business Impact: focuses on the actual resultsachieved by the programme participants as they success-fully apply what they have learned. Typical level 4 mea-

sures include output, quality, cost, time, and customersatisfaction. Although the programme may produce ameasurable business impact, there is still a concern thatthe programme may cost too much.Level 5, Return on Investment: compares the monitorybenefits from the programme with the programme costs.Although ROI can be expressed in several ways, it isusually presented as percentage or cost/ benefit ratio.

Measures pertaining to levels 1 and 2 are mostly quanti-tative and can be assessed easily at individual level. Levels 3and 4 involve task and organisation oriented measures

which are mostly investigated qualitatively [20]. It is clearthat it is necessary to measure programme effectivenessbeyond individual performance demands, involving the set-ting of clear qualitative measures, which is not easy toachieve.

Phillips [18] writes convincingly on the relationshipbetween competence and ROI. He is a devotee of develop-ing algorithms for the calculation of real financial benefit inorder to compare benefits with training, education, anddevelopment investment costs [21]. He argues that organi-sations have moved from training for activity to trainingwith focus on bottom-line results. However, it is verydifficult to see how project management development

programmes with complex inputs and slow-to-developintangible benefits can be fitted into this quantitativeapproach [21].

Industrial organisations prefer the concept of ‘‘BenefitMetrics”. Companies invest in training relating to manage-ment skills without a clear understanding of how knowl-

edge learned on a particular educational programmebecomes meaningful in practice, and benefits the organisa-tion as a whole. Most recently, researchers [22–24] havealso begun to question this relationship. ROI can bedefined as a subset of Benefit Metrics. The ROI approachis limited to the quantification of benefits from trainingand education, whereas Benefit Metrics attempts to investi-gate programme effectiveness by accounting for both tangi-ble and intangible educational benefits.

Benefit Metrics can best be explained by taking a threedimensional perspective. Organisations in partnershipswith educational institutions invest in educating and train-ing their employees in an attempt to seek improvements at

individual, project, business unit and corporate levels. Thecompanies expect improved results in terms of intangiblebenefits and the organisation’s bottom line, such as returnon capital employed (ROCE), profitability and cost sav-ings. Higher Educational Institutions are equally interestedin assessing programme effectiveness and efficiency in meet-ing corporate objectives [25]. The construct is illustrated inFig. 3.

Some would argue that there is no valid and reliablerelationship between indirect measures of intangiblebenefits like increased job satisfaction and increasedorganisational commitment and quantitative financial

‘‘bottom-line”

indicators. There may be arguments basedon anecdotal evidence, but these are not statistically validassociations [21]. Rowe [26] argues that it is not possibleto make measurements that enable ROI to be evaluatedwith respect to intangible benefits. On the other hand,direct measures evaluating the success of an organisationare quantitative and internationally understood (i.e. returnon capital employed (ROCE), profitability, turnover, andmarket share).

Measuring the effectiveness of a professional develop-ment programme requires the separation of training andeducation from a multitude of other variables (such as mar-keting, management attention, changes in procedures,

Organizations Educational Institutes/training

BM

Employees

Benefits---Increased Profits,ROCE, intangible benefits

Learning -----Increase incompetencies, skills development,knowledge gain

Fig. 3. BM Construct [25].



5/15

adjustment in standards, interest rates, economic and polit-ical stability) that can influence long-term organisationalperformance. The immediate effects of training and educa-tion on KPIs are usually invisible and they only becomevisible when monitored over long periods [27]. Fig. 4 pre-sents a framework for the investigation of Benefit Metrics

involving linkages and associations between investment,education and training and ROCE in relation to compe-tence, change, and their measurement.

4. The case of an industry-led modular masters project

management professional development programme

(PMPDP)

The method and speed of development of the syllabus,content, learning outcomes, form, delivery, managementand internationalisation of the programme give an indica-tion of how industry and academia can work swiftly andeffectively to meet a growing need for modular masters

level flexible blended learning. In September 1999 the leadindustrial consortium training partner (Rolls-Royce) con-cluded tendering for a training partner and in May 2000delegates (students) were studying a new modular mastersprogramme (PMPDP).

In 1999 a group of senior project management practitio-ners from across Rolls-Royce were meeting regularly withthe following objectives:

to create a community of project management; to share good practice; to encourage professionalism;

to develop project management staff at all levels; to raise the profile of project management and its impor-

tance to business performance across the company.

The programme objectives developed collaboratively byThe University of Manchester (UoM) and the industry

consortium were to support the above corporate objectivesby:

providing project management education at a postgrad-uate level;

providing a basis for a ladder of opportunity for those

seeking a postgraduate qualification; providing through the Foundation Module the prepara-

tion for sitting Association for Project Managementexaminations;

providing single modules in specific project managementfields;

providing flexible learning to enable busy engineers,managers and other employees to study for a postgrad-uate qualification;

An 8 module 12 topic matrix was developed todescribe the purpose and content of the taught modules.Currently this matrix (now Revision 19) is still used to

review the programme and forms an appendix to the con-tract between the consortium and the university. Duringthe run up to delivering the programme several majorreview meetings, attended by industry specialists and aca-demics, were held to progressively freeze the learning objec-tives, content and delivery. It is clear, looking back, thatthese meetings practitioner/academic balance to contentand helped to support a team spirit and common languagebetween academia and industry. The learning outcomesflowed from the interpretation of the discussions to developthe syllabus or Matrix.

Fig. 5 illustrates the relationship between curriculum

development, delivery and feedback. This is a seamless pro-cess involving delegates and their employers. Delegate feed-back occurs at every stage of delivery and assessment andfrom employers through the formal Steering Group andinformally via the project management single point contactbetween the consortium representative and the universityProgramme Director. The effectiveness of this project man-agement style is very important in terms of managingexpectations for all the stakeholders. All feedback isreported in some shape or form at the Steering Groupwhich meets twice per annum.

Central to the cycle illustrated in Fig. 5 are the learningoutcomes for the individual modules and the programmeas a whole. These learning outcomes are the basis of con-tent, delivery and assessment.

5. Programme content

For each module, delegates receive a workbook inhardcopy, CD or CD-Rom format, replicated on the uni-versity website. At each entry point (twice per annum)there are introductions explaining the objectives andlearning points for the modules. The modules includeself-assessment exercises and exercises designed to developcertain intellectual skills and reinforce theoretical concepts

and practical applications. Each module is accompaniedFig. 4. Benefit Metrics framework (ROI cycle) [21].



6/15

by a set text and directed reading. These are designed toform a personal project management library for delegates.Further reading and useful internet resources are listedtoo.

6. Assessment

All modules are assessed over the six months study per-iod and during the following plenary event with a two hourwritten examination where appropriate. Also, an industry

aligned coursework assignment is submitted. The pro-

gramme is moving towards the increasing use of continu-ous assessment and the incorporation of reflective practice.

7. Method of delivery

The programme is run as a web-enabled distance learn-ing programme. Delegates attend three day plenary ses-sions in April and October and one day mid-sessionevents in December, January, July and September. Plenaryevents cover orientation, lectures, keynote lectures, intro-

ductions, learning skills, clinics and examinations. Mid-ses-

DEMANDIndustry

WRITINGModuleCo-ordinators

OUTCOMESyllabus inform of Matrixof Modules

OUTCOMESModule WorkbooksSpecifications: Programme andModulesDelegate and Tutor GuideVirtual Learning Environment

DISCUSSIONUniversity and

Industry

INDUSTRYDRIVERSBehavioursCompetenciesKPIsReturn onInvestment

ObjectivesFields of studyProcess/DeliveryContentAssessmentPlanned Learning Outcomes

EDITINGUniversity andIndustry editing

DELIVERYResidential eventsPersonal learningWeb EnabledLearning

FEEDBACKFromDelegatesTutorsIndustrialists

ASSESSMENTCourseworkExaminations

Student Learning Process

OUTCOMESAction listChanges:to processto modules

STEERINGGROUPUniversityIndustry

To achieve LearningOutcomes

Fig. 5. Programme development and review cycle [28].



7/15

sion events cover tutorial support, seminars, clinics andrevision classes. All events involve supply chain integrationnetworking and informal events at which consortiumseniors are present. The companies play an important rolein approving the structure and content of the events andactively participate in their delivery.

8. Managed learning environment

UoM uses software called ‘‘WebCT” to facilitate a man-aged learning environment within which a virtual learningenvironment is sustained. The Joint Information SystemsCommittee (JISC) has produced a very useful BriefingPaper on Virtual and Managed Learning Environments(Everett, 2001).

The three functions of a managed learning environmentare:

content communication administration

The use of the software to web-enable the programme isgradually evolving and the feedback from delegates is cru-cial to the success of the web-enabled aspect of the pro-gramme. Delegates also have access to the UoM library,enabling access to thousands of on-line journals. Delegatescan access their local academic libraries, as reference onlyusers.

9. Management of the programme

The programme is managed on a day-to-day basis byUoM academic and academic related-staff. A ProgrammeManager is supported by a Distance Learning Adminis-trator. The eight modules are tutored and examined byModule Co-ordinators. Overseeing and managing theteam is the c Programme Director. The lead industrialpartner (Rolls-Royce plc) has a single point of contactmanager (Industrial Partners’ Manager) representing allindustrial partners in the consortium. He is in very closeregular contact with the Programme Director. The pro-gramme is subject to the normal academic scrutiny andquality assurance procedures including the involvementof an External Examiner. Research and Reflective Prac-tice industry led dissertation projects are supervised byuniversity staff in collaboration with industrial partnermanagers.

10. International developments

In November 2001, just after 9/11, the masters pro-gramme was launched in the United States for NorthAmerican employees of the industrial partners. Pennsylva-nia State University is running a common sister mastersprogramme with the same learning outcomes. This involves

close collaboration between both universities via a Memo-

randum of Understanding and Intellectual PropertyAgreement.

11. Benefit metrics, project management competencies and

learning outcomes

Considering the emerging body of research and debateon competence in project management, the strongest argu-ment for an association between measurable ‘bottom-line’indicators and intangible benefits from the activities of edu-cation, training programmes may be to do with compe-tence [21].

12. Competence

The argument for improving the competence of employ-ees is strong. A competence based approach enables theidentification of appropriate people to train and educate

[28,29]. Competence can be defined as [30]:

an underlying characteristic of an individual that iscausally related to criterion-referenced and/ or superiorperformance in a job or situation.

It is a deeply rooted and enduring part of personality,and can predict behaviour and performance in a varietyof situations. Moreover, it encompasses the ability to pre-dict individual performance against a specific standard.There are argued to be five characteristics of competence[30]:

(a) Motives: The things a person consistently thinksabout or wants that cause action. Motives drive,direct, and select behaviour towards certain actionsor goals.

(b) Traits: Physical characteristics and consistentresponses to situations or information.

(c) Self concept: A person’s attitudes, values, or self-image.

(d) Knowledge: Information a person has in specificfields.

(e) Skills: The ability to perform a certain physical ormental task.

13. Project management competence

The International Project Management Association(IPMA) [31,21, p.1095] undertook a benchmarking studyacross all members of the IPMA and developed the Inter-national Competency Baseline (ICB) which defines compe-tence as:

Knowledge þ Experience þ PersonalAttitudes

Here knowledge and experience relate to function and atti-tudes to behaviours. The IPMA definition of competence isvery much aligned with the views held by Crawford [32]

that competence is not a single construct.



8/15

The type and level of competence has practical implica-tions for a project manager’s development. As shown inFig. 6, knowledge and skills competence tend to be visibleand relatively, surface characteristics of people. They arerelatively easy to develop - professional development pro-grammes being the most obvious way. Self concept, trait

and motive competence are more hidden, deeper and cen-tral to personality. These types of competence are at thebottom of the ‘‘iceberg ” and are more difficult to assessand develop. It is more cost-effective to select for thesecompetencies, rather than to train. Self-concept competen-cies, comprising attitudes and values, lie somewhere inbetween and can be changed through personality develop-ment experiences, albeit with more time and difficulty [30].

The literature identifies three different approaches todefining project management competence [29].

(1) The input approach, common in the USA, assumes thatindividuals require knowledge, skills and behaviours

to be competent at work. This approach forms thebasis of the BoK developed by the PMI in theirPMBoK [29].

(2) The UK based process approach accounts for pro-cesses and functions needed by project managers todeliver projects successfully. The IPMA’s Interna-tional Competence Baseline is derived from thisapproach [29].

(3) The output approach popular in Australia focuses onthe actions of project managers to deliver projects.This approach forms the baseline for the project man-agement competence standard developed by AIPM

[29].

Concern for the competence of project managers hasfuelled interest in the development of standards and thecertification of project managers, used for assessment, rec-ognition and as a guide for the development of projectmanagement competence. Standards are derived fromtypical activities, responsibilities and requirements from

practice. For example the IPMA [21, p.1095] establisheda four level structure to assess PM competence.

IPMA Level A: the certified project director shall be ableto direct all projects of the company.

IPMA Level B: the certified project manager shall be able

to manage complex projects him/ herself.IPMA Level C: the certified PM professional shall be ableto manage non-complex projects him/her-self and assist the manager of a complexproject in all aspects of projectmanagement.

IPMA Level D: the certified PM practitioner shall have theknowledge in all project managementareas.

Professional project management competencies areachieved by the combination of education and knowledgeacquired during training and skills developed through

experience and application of such knowledge and experi-ence through effective behaviours [28].

14. Measuring competence

There are several indicators commonly used in the mea-surement of competence [21, p.1099]: continuous educa-tion, examinations, portfolios, self–assessments,interviews, outcomes and peer review. Three techniquescan be used to measure competence: checklists, observa-tional methods and the framework approach. The firsttwo approaches focus on performance measurement

whereas the framework approach relates to an integrativeperspective on competence.

A 360-degree feedback approach is widely used for mea-suring project management competence [21, p.1100]. Dataare collected from an individual’s peers, direct reports,internal customers, and line managers. The techniqueinvolves questionnaires (sometimes software based) toobtain an evaluation of individuals from multiple sources.

Skill

Knowledge

Self concept

Trait,Motive

Attitudes/ Values

Core Personality: Dfficult to developi

Surface competencies: Mosteasy to develop

SkillKnowledge

Self-Concept Trait Motive

Hidden

Visible

Fig. 6. Competence Structure [30, p.11].



9/15

Companies like Rolls-Royce, Goodrich, AMEC andEDS have an interest in measuring the competence of theirproject management communities. Rolls-Royce, for exam-ple, has developed a web-enabled competence measure-ment system [33].

Whatever the method of education or training adopted

it is important to have clarity on the objectives. Assessmentneeds to be aligned with learning outcomes.

15. Learning outcomes (LOs)

British universities are required by the Quality Assur-ance Agency for Higher Education (QAAHE) to state theLearning Outcomes (LOs) for all programmes and modulesof programmes, at both undergraduate and postgraduatelevel.‘‘Reduced to its simplest form, an outcomes-basedapproach to learning has three components:

an explicit statement of learning intent expressed as out-comes that reflect aims and values;

the process to enable the outcomes to be achieved anddemonstrated (curriculum, teaching, learning, assess-ment and support methods);

the criteria for judging achievement of the intended out-comes.” [34, Appendix D]

These learning outcomes are subdivided into fourelements:

Academic Knowledge

Intellectual Skills Subject Practical Skills Transferable Skills

The APM, PMI and other organisations concerned withproject management publish BoKs. These were consideredwhen developing the case study programme presentedabove, but the BoKs do not easily relate to either learningoutcomes or competencies. Further, there are some obvi-ous gaps in all the BoKs; particularly in the area of peopleand culture.

There is no universal agreement or even similaritybetween BoKs and what industry, academia, and profes-sional bodies and associations believe to be the key compe-tencies, learning outcomes and knowledge sets. This maybe a good thing because in the interest of critical debate dif-ferent perspectives are necessary. Therefore, attempts tounify BOKs may be actually detrimental to a real under-standing of the issues and critical thinking.

Organisations and their managers want to define andmeasure competence. In order to promote and developthe competence of project managers, education is clearlynecessary.

No clear linkage exists between competencies, LOs,BoKs and BM. LOs may, in part, be measured by compe-

tencies and the desire by industry for particular competen-

cies may lead to the specification of certain LOs. A numberof groups (universities, professional bodies, employers)have vested interests in promoting different positions oncompetence, BoKs and LOs. In practical terms, in purchas-ing education and training, industrial organisations expectthe effect on students to lead to improvements in individual

and team performance; contributing in turn to the profit-ability of the organisation.

16. A brief discussion on research being undertaken into

benefit metrics

Three consortium funded studies being undertaken bythe authors are introduced below. The first used qualitativetechniques to ascertain the effectiveness of the case studyprogramme mentioned above (PMPDP). The other twostudies used quantitative techniques. These studies formthe beginning phase of a larger research project investigat-

ing BM.The first study [35] used the Return On Investment(ROI) methodology proposed by Philips [18] to investigatethe key objective of the Programme; that is to improve thecompetence of employees. The model comprises 4 stages:(a) evaluation planning; (b) data collection; (c) data analy-sis; (d) programme evaluation. Qualitative data were gath-ered from delegates via a questionnaire survey.

The second and third projects [33,36] used the Rolls-Royce Knowledge and Experience questionnaire (RR-KEQ) [37] (an automated computer based electronic sys-tem attached as Appendix A) that facilitates the evaluationof a person’s knowledge and experience in the topics thatmake-up the APM BoK. Data in the form of RR computedknowledge and experience grades were taken from the RR-KEQ to compare the competence of delegates at differentstages of the programme with a control group which hadnot undertaken any comparable professional developmentin project management.

17. Sampling

Employees participating in the PMPDP were dividedinto cohorts based on when they began their studies. Thefirst five cohorts comprising 83 delegates were chosen asa working population because they had significant experi-ence of the programme (response rate 25 per cent).

In the second survey, 358 completed RR-KEQs wereavailable, of which 55 had been completed by delegateson the PMPDP, leaving 303 questionnaires to act as a con-trol group. The working population for the third projectcomprised 181 employees who completed the RR-KEQ in2005. Among them, 65 were delegates who had studiedPMPDP modules.

The population for all the three surveys covered differ-ent business units and management levels. Data were strat-ified in terms of IPMA project management role levels (A,

B, C and D).


http://-/?-http://-/?-


10/15

18. Qualitative analysis

In the first survey the questions asked were:

(1) Give the title or descriptor of an area of benefit.(2) To what extent is this benefit attributable to study on

the PMPDP? (3) In what way did your learning experience on thePMPDP relate to the benefit reported here?

Twenty one responses to question 1 indicated areas of benefits attributable to the PMPDP (Fig. 7).

The analysis of question 1 reveals that all modules of PMPDP (apart from module 1, Foundation) are notedby at least one delegate as being of benefit. The secondquestion ascertained the extent to which delegates believedthat improved performance was due to studying on thePMPDP (Fig. 8).

We could say that those areas where the PMPDP is saidto be making a small contribution are the areas whereemployees have sufficient knowledge to carry out a partic-ular task already, and hence do not need the additionalknowledge. In other words the PMPDP is training employ-ees for more complex tasks that they may need to under-

take in the future. It could be argued that these areasneed to be re-evaluated so that they are better aligned withthe needs of the employee, such as configuration and pro-gramme management.

Delegates were also asked to identify an area of projectmanagement in which they believed they need the mosttraining; i.e. the area which causes them the most problems.Table 1 shows these problem areas and the frequency withwhich they have been selected.

Most of these problem areas can be related to thoseselected in relation to benefits attributed to studyingPMPDP. Table 2 shows these areas along with the fre-quency with which they were highlighted as problem areas,

and the percentage contribution of the PMPDP in theseareas. Table 2 aims to show whether or not the most fre-quently specified problem areas are also those which aresaid to be benefiting most from PMPDP study.

There is one PMPDP and three non-PMPDP areas thatdo not have a match. The ‘‘People and Culture” module, inparticular, is intended to be useful in many different prac-tical contexts, so whilst people and culture issues are nothighlighted specifically, this module probably contributesto in solutions in several areas.

The matter of ‘‘Inexperience” is one that cannot besolved by any taught aspect of the PMPDP. It may be

the case that participation in a learning programme willlead to employees paying more attention to increasingtheir experience. For example, looking for areas wherethey can implement a new tool or technique taught on

Frequency of selection by delegates of

areas which benefit from the PMPDP

17.2%

3.4%

10.3%

10.3%

6.9%6.9%10.3%

10.3%

3.4%

13.8%

6.9% Commercial & Procurement

Configuration Management

Cost

Organisation

People

Planning

Programme management

Project management

Quality management

Risk analysis

Strategy

Fig. 7. Frequency of selection by delegates of areas which benefit from thePMPDP [35].

Average percentage contribution of PMPDP

to each area of benefit

70.0%

20.0%

48.3%

75.0%

90.0%

42.5%

27.5%

61.7%60.0%

62.5%

75.0%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

PM areas

P e r c e n t a g e

Commercial & Procurement

Configuration Management

Cost

Organisation

People

Planning

Programme management

Project management

Quality management

Risk analysis

Strategy

Fig. 8. Average percentage contribution of PMPDP to each area of benefit [35].



11/15

the programme. There is some anecdotal evidence from theconsortium that supports this claim.

In question 3, individuals were encouraged to describehow the PMPDP had contributed to their work.

Responses to this question indicate that the PMPDP con-tributes to the improvement of all types of competence. Interms of knowledge, individuals reported improvements intheir negotiation, communication and conflict resolutionskills. Also, the new tools and techniques learned fromthe PMPDP led to better bids to clients, better risk assess-ments, reduced cost over-runs and earlier damagedetection. In the area of behavioural competence,improvements were noted in relation to levels of confi-dence in ability and skills. Improved confidence was seento lead to improved on-the-job performance and as wasmentioned previously, experience can not be taught.PMPDP was reported to have led to a more active and

closer project management community likely to lead toimprovements in experience.

19. Quantitative analysis

In surveys 2 and 3, mean knowledge scores against eachmanagerial level were calculated on the basis of the partic-ipation of delegates in different modules of the programme.

Since the data contained no numeric values to measure theknowledge and understanding of each respondent, equiva-lent numerical mean scores were assigned to the RR, non-numeric, knowledge grades. This equivalence is shown inTable 3.

The mean scores of each individual in the control

group were also calculated to obtain an overall meanknowledge score against each managerial position. Table4 quantifies the effects on mean knowledge scores of del-egates studying PMPDP modules, and also compares theeffects on mean knowledge scores of PMPDP delegateswith non-PMPDP employees. This comparison is graphi-cally represented in Fig. 9. The same analyses were car-ried out for survey 3. The final results are graphicallyrepresented in Fig. 10.

The results of surveys 2 and 3 indicate that, on comple-tion of the PMPDP, an increased level of knowledge com-petence could be expected at all managerial levels. Whenthe UoM computed mean knowledge scores (see Table 3

for equivalent RR non-numeric grades) for each manage-ment level were compared with the IPMA recommendedcertification levels, it became evident that by the timedelegates had completed 6 modules on PMPDP, it couldbe expected that they either meet or exceed the IPMA cer-tified level for that management position. It appears from

Table 1Problem areas in project management [35]

Problem Area Number of selections (N) Frequency (%)

Project Definition 16 7.69Resources 38 18.27Organisation 73 35.10Time 37 17.79

Costs 11 5.29Quality 1 0.48Risk 4 1.92Contracts 12 5.77Change 13 6.25Inexperience 3 1.44

Table 2

Problem areas in project management and contribution of PMPDP [35]Problem Area Corresponding area of PMPDP Frequency of selection of problem area Contribution of PMPDP

Project Definition Project management 7.69% 61.7%Resources Planning 18.27% 42.5%Organisation Organisation 35.10% 75.0%Time Planning 17.79% 42.5%Costs Cost 5.29% 48.3%Quality Qty management 0.48% 60.0%Risk Risk analysis 1.92% 62.5%Contracts Commercial & Procurement 5.77% 70.0%Change Strategy 6.25% 75.0%Inexperience None applicable 1.44% -None specified Configuration Management - 20.0%None specified People - 90.0%

None specified Programme management - 27.5%

Table 3RR knowledge grade/UoM computed mean score equivalence [33]

RR knowledgeGrade

UoMComputed

Mean Score

RR knowledgeGrade

UoM ComputedMean Score

A+ 14.5 C 6.5A 13.5 D+ 5.5A 12.5 D 4.5B+ 11.5 D 3.5B 10.5 E+ 2.5B 9.5 E 1.5C+ 8.5 E 0.5C 7.5



12/15

analysis that nearly all PM practitioners and professionalsexceeded their recommended IPMA certification level (Dand C respectively) upon completing 6 PMPDP modules.However, the group that showed the greatest improvementwas the Project Managers (IPMA level B). Seventy twopercent of PMs not on the programme fell below therequired IPMA certification level but all PM delegates

met or exceeded the certified level upon completion of 6PMPDP modules.

Similar results were drawn from survey 3. All PM prac-titioners and professionals were found to exceed their rec-ommended IPMA level after completing 6 modules. Inthe case of project managers, 25 per cent met the recom-mended IPMA level B and 75 percent exceeded it. In com-parison with non-PMPDP PMs, 36 per cent fell below, 32

per cent met and 32 per cent exceeded their IPMA certifiedlevel.Generally, it appears from the results that we can

hypothesise that there may have been a general improve-ment in competence throughout the organisation due todelegates studying the PMPDP.

20. Research limitations

There are certain limitations to both qualitative andquantitative approaches for measuring the effectivenessof a professional development programme. The qualita-

tive study carried out in the first survey aimed to measurethe effectiveness of PMPDP using the ROI methodology[18]. The ROI model comprises 4 levels and demandscomplete evaluation of a management development pro-gramme by using appropriate research instruments (ques-tionnaires, focus groups and interviews) at every level.This research project is ongoing and at this stage onlyone questionnaire has been fully developed; primarilyinvestigating level 4 of the ROI model (business results).Secondly, the model suggests the inclusion of statementsand estimations from delegates, graduates, supervisors,line mangers, customers and experts. However, the train-ing evaluation undertaken in this survey is based on theview points of delegates only. However, the data appearto be rich. Delegates are those individuals who can seefirst hand the positive and negative aspects of the profes-sional development programme. Therefore, the opinionsof this group can be considered meaningful in measuringthe effectiveness of the PMPDP.

The small sample sizes are a limiting factor. However,this research is ongoing and larger sampling will be con-ducted. The interim findings of survey 1 are based on 21respondents representing 25 per cent of the working popu-lation. The results of survey 2 are statistically verifiedthrough chi-squared analysis. However, there are limita-

tions to this study, due to the very nature of quantitative

0

2

4

6

8

10

12

14

M e a n k n o w l e d g e s c o r e s

P r a c

t i t i o n

e r

P r o f e s s i o n

a l

M a n

a g e r

T o t a l A

v e r a g e

No of modules

Comparison of PMPDP and non-PMPDP employees

Non-PMPDP PMPDP Module 1 PMPDP Module 8

Fig. 9. Survey No.2 [36].

Comparison of PMPDP and Non-PMPDP employees

0

2

4

68

10

12

14

P M P r a c t i t i o

n e r D

P M P r o f e s

s i o n a l C

P r o j e

c t M a

n a g e

r B

A l l r o

l e s ( B

, C , D )

No of Modules

M e a n K n o w l e d g e s c o r e s

Non-PMPDP PMPDP Module 1 PMPDP Module 8

Fig. 10. Survey No.3 [33].

Table 4Comparison of PMPDP and Non-PMPDP employees (Survey No.2) [36]

Position Participation level

PMPDP module 1 Mean Score PMPDP module 8 Mean Score Non-PMPDP Mean Score

Practitioner D C+ 8.5 B+ 11.5 C 7.5Professional C C+ 8.5 B+ 11.5 C+ 8.5

Manager B B

9.5 A

12.5 B

9.5Overall Score C+ 8.5 B+ 11.5 C+ 8.5



13/15

research techniques and the RR-KEQ itself. The RR-KEQis a facilitated tool and therefore validity and reliability arequite strong for this type of instrument. Bias may be posi-tive when the employee is over-confident and assigns him/herself a higher score, or negative due to negative attitudestowards work. In survey 3, 17 per cent of the RR-KEQs

submitted were facilitated by managers and 7 percent bypeers.

21. Conclusions

The mode of delivery of the case study programme(PMPDP), described in this paper, is not complex. Acombination of intensive residential events coupled withtraditional and web enabled blended learning appearsto provide a robust learning and professional develop-ment platform. The linkage between all the stakeholdersof the programme: students (delegates), their line manag-ers, senior managers, academic staff and academic related

support staff is maintained through a project managedapproach to this academic-industry partnership. The sin-gle point of contact between the industrial consortiumand the university provider is an important feature of this, together with the influence of an active steeringgroup with delegate representation. During the inceptionand scope definition stages of the ‘‘project” team build-ing occurred as a result of a common purpose withrespect to the development of learning outcomes arisingfrom industry led objectives for the programme. Argu-ably, this is unusual in comparison to the ‘‘normal” rela-tionship between academia and industry in which the

investment of specialist project management resourcefrom industry whilst possible may not usually be madeavailable due to considerations of cost and project man-agement capability from a university provider, althoughdesirable, is frequently impossible. The project managedstability of project management development does notcome cheap and effectiveness must be measured. Theprogramme requires the understanding and management

of expectations for all stakeholders in the context of apartnership.

The serious consideration of Benefit Metrics (Return onInvestment) is only possible if both qualitative and quanti-tative sensitive data are available. This is not simple andrequires significant resources. Therefore, without project

managed partnerships, trust and commitment it is difficultto see how this could be achieved.The general trend in all the three surveys indicates that

PMPDP is contributing positively in increasing the knowl-edge of delegates. It appears from the analysis that the PMknowledge of employees participating in the PMPDP hasgradually increased as they study more PMPDP modules.This is found true for all managerial positions.

It is further concluded that studying the PMPDP doeshave a positive effect on project manager’s knowledgeand their profiles in relation to the IPMA levels(A,B,C,D). Although, it cannot be determined that thePMPDP actually causes an improvement in the knowledge

of employees, it appears from the analysis that employeesattending the PMPDP would finish with a higher knowl-edge level compared to when they started. Hence, it canbe deduced that professional development programmes likePMPDP provide an effective way of ensuring improve-ments in employee competence. It is further observed thatthe PMPDP does successfully address most of the identifiedareas requiring educational focus in particular projectmanagement areas.

It appears from the results of all the three surveys thatprofessional development programmes like the PMPDPcan contribute to the development of competence and

competencies in individuals, leading to benefits for theorganisation as a whole. The results are in accordancewith the views of a number of published articles on thissubject that professional development programmes doplay a significant part in improving the competence of employees. However, they also contradict other views [5]that competence is difficult to quantify and cannot bemeasured.



14/15

Appendix A. Rolls-Royce Knowledge and Experience

Questionnaire (RR-KEQ) sample [37]



15/15

References

[1] Avots I. Why does project management fail? Calif Manage Rev1969;12:77–82.

[2] Munns AK, Bjeirmi BF. The role of project management in achievingproject success. Int J Project Manage 1996;14(2):81–7.

[3] Project Management Institute. A Guide to the Project Management

Body of Knowledge, PMI, 1996.[4] Dixon M. APMBoK, APM UK., 2000.[5] Crawford L. Profiling the Competent Project Manager. Proceedings

of PMI Research Conference, 21-24 June, Paris, France, 2000, p. 3-15, PMI.

[6] Atkinson R. Project Management: cost, time and quality, two bestguesses and a phenomenon, its time to accept other success criteria.Int J Project Manage 1999;17(6):337–42.

[7] Morris, P.W.G. and Hough, G.H. The preconditions of success andfailures in major projects (Technical Paper No.3). Oxford: MajorProjects Association, 1986.

[8] Lim CS, Mohamed ZM. Criteria of project success: an exploratoryexamination. Int J Project Manage 1999;17(4):243–8.

[9] Shenhar AJ, Dvir D, Levy O, Maltz AC. Project success: a multi-dimensional strategic concept. Long Range Planning

2001;34(6):699–725.[10] Pinto JK, Slevin D. Project Success: Definitions and Measurement

Techniques. Project Manage J 1988;19(1):67–72.[11] Freeman M, Beale P. Measuring project success. Project Management

Journal 1992;23(1):8–17.[12] Shenhar AJ, Levy, et al. Mapping the dimensions of project success.

Project Manage J 1997;28(2):5–13.[13] Baccarini D. The logical framework method for defining project

success. Project Manage J 1999;30(4):25–32.[14] Wang X, Huang J. The relationships between key stakeholders’

project performance and project success: Perceptions of Chineseconstruction supervising engineers. Int J Project Manage2006;24:253–60.

[15] Rowden R, editor. Workplace learning: Debating five criticalquestions of theory and practice. San Francisco: Jossey-Bass; 1996.

[16] Daley JB. Learning and Professionals Practice: A study of fourprofessions. Adult Education Quarterly 2001;52(1):39–54.

[17] Hinds J. Professional Development. Financ Times 2004:1.[18] Phillips J. Return On Investment in Training and Performance

Improvement Programmes. Burlington, MA, USA: Elsevier Science.;2003.

[19] Kirkpatrick DL. Techniques for evaluating training programmes.Training Dev J 1979;33(6):78–92.

[20] Thiry M. How can the benefits of PM training programmes beimproved? Int J Project Manage 2004;22:13–8.

[21] Gale AW. Competencies: Organizational and Personal. The WileyGuide to Managing Projects. New Jersey, USA: John Wiley and Sons,Inc; 2004.

[22] Black R. and Schell J. Learning within a situated cognitionframework: Implications for adult learning.Paper presented at theAmerican Vocational Association Convention, Denver, CO. (ERICDocument Reproduction Service No. ED389 939), 1995.

[23] Eraut M. Developing professional knowledge and competence.Washington DC: Falmer; 1994.

[24] Grzyb S. Effects of organizational role and culture on participation incontinuing professional education. Paper presented at the AnnualMeeting of the American Educational Research Association, Chi-cago, IL. (ERIC Document Reproduction Service No. ED409 284),1997.

[25] Alam M. Benefit Metrics Research Project: 1st Year Progress Report(unpublished), The University of Manchester UK. 2006, 268.

[26] Rowe C. Assessing the effectiveness of Open Learning: TheBritish Aerospace Experience. Ind Commercial Training1994;26(4):22–7.

[27] Boverie P, Mulcahy DS, Zondlo JA. 1995. Evaluating the effective-ness of Training Programmes.” [Online] Retrieved 251206, fromhttp://www.zondlo.com/access/eval.htm.

[28] Gale AW, Brown M. Project Management Professional Develop-ment: An industry led programme. J Manage Dev2003;22(5):410–25.

[29] Song S. 2006. Developing Project Managers in China (PhD Thesis).MACE, Manchester UK, The University of Manchester.

[30] Spencer LM, Spencer SM. Competence at work. NY: John Wiley &Sons; 1993.

[31] IPMA. International Project Management Association: InternationalCompetence Baseline (version 3). ISBN 0-9553213-0-1, IPMA, 2007.

[32] Crawford L. Senior Management perceptions of project managementcompetence. Int J Project Manag 2005;23:7–16.

[33] Alam M. Rolls-Royce Knowledge & Experience QuestionnaireAnalysis 2005 (Research Report), (unpublished), The University of Manchester UK, 2006.

[34] QAAHE. Handbook for Academic Review, Appendix D. QualityAssurance Agency for Higher Education, available at www.qaa.or-g.uk., 2002.

[35] Tsopela, M. Return On Investment of the Project ManagementProfessional Development Programme (PMPDP) (MSc dissertation).MACE. Manchester UK, The University of Manchester, 2003.

[36] Statter, C. 2004. The development of functional and behaviouralcompetence in relation to success amongst project managers (Under-graduate Research Project). MACE. Manchester UK, UMIST.

[37] Piling N. Rolls-Royce project management competency assessmenttool. Derby, UK: Rolls-Royce; 2006.

http://www.zondlo.com/access/eval.htmhttp://www.qaa.org.uk/http://www.qaa.org.uk/http://www.qaa.org.uk/http://www.qaa.org.uk/http://www.zondlo.com/access/eval.htm

dev and delivery of industry led proj mgt project management education

Documents