research measurement for realizing quality & excellence in universities

Research Measurement for Realizing Quality & Excellence in Universities

24 April 2012, Majmaah University, KSA

Deanship of Quality and Skills DevelopmentSeminar

Tariq A. Aldowaisan, Ph.D., CSP, CQM/OE, CQE, CQAAssociate Professor, Kuwait University (KU)

MD, Gulf Lead Consultants (GLC)

أساسية البحثيخطوة قياسالنشاطجامعاتنا في والتميز الجودة لتحقيق

Copyright@ 2011 Kuwait University & Gulf Lead Consultants, All Rights Reserved. 2

1. Provide an overview of the meaning of quality; the relationship between quality and excellence; and their applications in the university context.

2. Recognize the importance of performance measurement in driving performance improvement.

3. Suggest the four-dimensional OECD performance measurement system, and give examples of its application in the research activity.

Seminar Objectives

http://3.bp.blogspot.com/-Kl3ibMEpNlM/T0Btk9jSy2I/AAAAAAAAAhM/Dq-y0kNWYBA/s1600/measuring+success.jpg


1. The meaning of quality

2. Quality obligations of universities

3. From quality to excellence

4. Importance of performance measurement

5. Performance measurement systems: Stakeholders, BSC, & OECD.

6. Applications of the OECD Model in research activity

7. Concluding recommendations on performance measurement in research

8. Overcoming implementation challenges

Agenda


Quality is customer satisfaction through:

• Capture of customer requirements – Quality of Design

• Compliance with customer requirements– Quality of Conformance

What is Quality?

Who are our c

ustomers?


What is Quality?

Quality is to

• identify and encode customer requirements through STANDARD/SPECIFICATIONS, and then

• achieve customer satisfaction by complying with STANDARD.

Diameter 12 ± 0.30 mm11.7 mm 12.3 mm

Defective Defective

Payment not to exceed 30 days

30 days

Defective

CUSTOMER

REQ

UIR

EMEN

TSHard Standards in Production


What is Quality?Hard Standards in Service


What is Quality?

Continual Improvement of the QMSContinual Improvement of the QMS

Input Outputvalue adding

activities

info. flow

info. flow

CUST

OM

ER R

EQU

IREM

ENTS

CUST

OM

ER S

ATIS

FACT

IONmanagementresponsibility

managementresponsibility

productrealization

productrealization

resourcemanagement

resourcemanagement

meas, analysisimprovement


productproduct

clause 4

clause 5clause 6 clause 8

clause 7

value adding activities

Management Systems – Soft Standards


• Examples from ISO 9001:2008 Quality Management System Standard:

– 8.2.1 Customer satisfactionAs one of the measurement of the performance of the quality management system, the organization shall monitor information relating to customer perception as to whether the organization has met customer requirement.

– 8.2.4 Monitoring and measurement of productThe organization shall monitor and measure the characteristics of the product to verify that product requirements have been met.

Management Standards


Hard & Soft Standards

Comparison Hard Soft

Type Technical Management

Language Specification Requirements/Criteria

Coding Numeral Shall/Must/Is/Are

Checking Inspection Audit

Difficulty of Checking Low High

Example BSI-999, CSE, ISO 17025

ISO 9001, CSE, ISO 17025, ABET


• Basic Quality Obligationgraduates’ abilities are aligned with the market’s current and potential needs and expectations.

• National Quality Obligationcontribute to the development of their nations.

Quality Obligations of Universities

Source: Aldowaisan, T. “Quality Obligations and Newly Established Universities”, Quality Beat, 1st Issue October 2011.


1. Alignment of the university’s strategic plan with the national plan.

2. Selection of a robust performance measurement system.

3. Choice and development of faculty members.

Quality Obligations of UniversitiesQuality Controls

Source: Aldowaisan, T. “Quality Obligations and Newly Established Universities”, Quality Beat, 1st Issue October 2011.


From Quality to Excellence

Customer Focus

Organization Focus

Meet Requirements

Exceed Requirements

Good operation(e.g. 3 )

Great operation(e.g. > 6)

CustomerResults

StakeholderResults

Employers

ResearchCommunity

SocietyFaculty& Staff Students

Ranking Bodies

Accredit Rank

AccreditationBodies

Satisfy Delight


BNQP – Performance Excellence Program

1Leadership

3Customer

Focus

7Results

4Measurement, Analysis, and Knowledge Management

2StrategicPlanning

6Operations

Focus

5Workforce

Focus

Organizational Profile:Environment, Relationships, and Strategic Situation

2011-2012 Education Criteria for Performance Excellencewww.nist.gov/baldrige/publications

120

85

85

90

85

85

450

Education Criteria for Performance Excellence Framework


BNQP – Performance Excellence Program

1Leadership

3Customer

Focus

7Results


2StrategicPlanning

6Operations

Focus

5Workforce

Focus


2011-2012 Education Criteria for Performance Excellencewww.nist.gov/baldrige/publications

120

85

85

90

85

85

450

Importance of Performance Measurement


ISO 9001 QMS

Continual Improvement of the QMSContinual Improvement of the QMS

Input Outputvalue adding

activities

info. flow

info. flow

CUST

OM

ER R

EQU

IREM

ENTS

CUST

OM

ER S

ATIS

FACT

IONmanagementresponsibility

managementresponsibility

productrealization

productrealization

resourcemanagement

resourcemanagement



productproduct

clause 4

clause 5clause 6 clause 8

clause 7

value adding activities

Importance of Performance Measurement


Excellence Drivers

1Leadership

3Customer

Focus

7Results


2StrategicPlanning

6Operations

Focus

5Workforce

Focus


Improvement Team

Education Criteria for Performance Excellence Framework


• Leadership – a priori, – 2nd weight (120)

• Results– clarity of target, – 1st weight (450)

• Improvement Team– Spearhead change– Link leadership & Results Triads

Determiners of Deployment Effectiveness


Results – Get it Right

An Encompassing Vision

• Times Higher Education• QS World University Ranking• Academic Ranking of World Universities• US News World Best Universities• etc.

Achieving success in our universities,www.glc-icm.com.

http://www.glc-icm.com/


Results – Get it Right

Real Goals – Outcome based

Research

Teaching

Community Development


Performance Measurement SystemsStakeholders’ Perspectives

PMsPMs

PMs

PMs

PMs


Performance Measurement SystemsBalanced Scorecard (BSC)

Source: SmartDraw 2010

PMs

PMs

PMsPMs


Performance Measurement SystemsOECD Model – Activity Based

Source: Measuring Government Activity, OECD 2009

Inputs (Resources) Activities Intermediate

OutcomesFinal

OutcomesOutputs

organization structure andmanagement provisions

Context


Performance Measurement SystemsOECD Model


OutcomesFinal

OutcomesOutputs

Efficiency

Productivity Quality

Customer

National/ University Objectives

Effec

tiven

ess


OECD Performance Measures Dimensions

• Efficiency: measures how well resources are utilized to produce the desired output (resource/input-driven).

• Productivity: measures the capacity to produce the maximal output possible for given resources (output-driven).

• Quality: measures how well the produced outputs meet or exceed relevant quality dimensions standards (output-focus).

• Effectiveness: measures how well the outputs and outcomes agree with the planned/targeted outputs and outcomes (outcome-focus)

Doing Things Right Doing Right Things


OutcomesFinal

OutcomesOutputs

organization structure andmanagement provisions

Context


• Teaching – Undergraduate

• Teaching – Graduate

• Research

• Community Services; e.g. consultation, professional development, and testing.

Key University Activities


The Research Activity

• Projects• Papers• Patents• Spinoffs

ResearchActivity

• Budget• Faculty

• Facilities

Country Priorities

Research Community

University Ranking

Business Community


• Efficiency: the cost of funded published papers (journal & conferences). The cost may be expanded to include the administration of the research activity.

• Productivity: the number of published funded and non-funded papers (journal & conferences) per faculty (all faculty). Of course more specific measures of productivity could be developed for journals and conferences separately.

• Quality: the percentage of papers (funded and non-funded) that are published in the top quartile of the Journal Citation Reports (JCR)

• Effectiveness– the compounded annual growth rate (CAGR) of papers (journal & conference) over

a certain period (Immediate outcome).– the percentage of papers addressing national priorities (Intermediate outcome).– the percentage of papers utilized by market (Final outcome).

Suggested Performance Measures for Research Activity


Performance Measures & Objectives

• Research Investment• Faculty members

25% address Priorities

At least 20% published in top quartile of JCR

Quality1.5 paper per faculty member

productivity

2000 KD per paperEfficiency

10% utilized by market

Note: 1. Papers include conference and journal papers as well as funded and non-funded.2. Faculty members include all faculty members (active and non-active).3. For Research Investment Efficiency, we consider funded papers only.

25% address priorities

30% CAGR

Effec

tiven

ess

Research Activity Papers


Using the PM to Drive Improvement

Manage to achievetarget value

Set target value &attainment date

Determine basevalue of PM

NoUndertake

corrective actionProgress

OK?

Yes

Claim success

Target valueattained?

No

Yes

1

2

34

5

"You can't manage what you

can't measure "

__ Edwards Deming


• Process/activity based

• More appropriate for non-profit institutions

• Readily supports ratio performance metrics

• Simple, efficient, robust

• Supports doing things right and doing the right things

Advantages of OECD Model


• Focus more on reporting outcome-related performances and their development over the year.

• Include non-funded published papers as well as conference papers to calculate faculty productivity (i.e. papers per faculty); which is an important measure of immediate outcome (i.e. output).

• Include a measure of research cost to calculate research investment efficiency (i.e. KD/paper).

• Include a priority effectiveness index related to research relevance to the priority areas (i.e. KD/KD or Papers/papers).

• Include a marketability effectiveness index related to marketability of research output (i.e. KD/KD or Papers/papers).

Recommendations on Performance Measures in Research


• Include commercial effectiveness indices on spinoff businesses and patents.

• Include a leading indicator faculty research engagement (i.e. percentage of faculty involved in research).

• Include a leading indicator on faculty collaboration index (KD of researches that involves two or more faculty of total research investment).

• Include a leading indicator on interdepartmental collaboration index (i.e. % of collaboration research projects involving collaboration from different departments).

• Include a leading indicator on large-scale interdisciplinary research collaboration (i.e. % of collaboration large-scale research projects involving different disciplines).

• reflect on the impact of the level and quality of research on the university rank; through engaging credible ranking bodies.

Recommendations on Performance Measures in Research


No. Measure OECD Perspective

BSC Perspective

Stakeholder Perspective

1 Faculty productivity

Productivity Internal Process

faculty, research community, Academic Institution

2 Investment efficiency

Efficiency Financial Funding agencies

3 Publications quality

Quality Customer Research community, faculty, Academic Institution

4 Publications growth

Immediate outcome – Effectiveness

Academic Institution

5 Priority effectiveness index

Intermediate outcome – Effectiveness

National and local community

Cross-Check for Completeness


No. Measure OECD Perspective

BSC Perspective

Stakeholder Perspective

6 Marketability effectiveness index

Final outcome – Effectiveness

National and local community, Academic Institution

7 Faculty research engagement

Learning & Growth

Faculty, Academic Institution

8 Faculty collaboration index

Learning & Growth


9 Interdisciplinary collaboration index

Learning & Growth


Cross-Check for Completeness


• Ensure selected performance measures are collectively comprehensive.

• Ensure that the selected performance measures are mutually exclusive.

• Gain the benefits of each measurement framework.

Benefits of Cross-Checking


Definition of a Performance Measure

•Base Value •Assumptions

•Data Sources

•Benchmarks•Targets

•Beneficiaries

•Analyses

•Measurement Unit

•Frequency

•Formula

•Dimension

•Interpretation Reference•Goal/Objective


• Leadership Commitment

• Culture – Fear of accountability– Not meeting a target construed as a failure

• Competency building– General awareness starting at top– Deep knowledge & skills of improvement team to

cover both measurement and improvement methods and tools

• Cumbersome to monitor– Automation is a must

Implementation Challenges


واالمتنان الشكر كل

الدويسان المحسن عبد طارق[email protected]

[email protected]

mailto:[email protected]

mailto:[email protected]

http://www.glc-im.com/

research measurement for realizing quality & excellence in universities

Documents